[jira] [Commented] (CASSANDRA-12423) Cells missing from compact storage table after upgrading from 2.1.9 to 3.7
[
https://issues.apache.org/jira/browse/CASSANDRA-12423?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15851069#comment-15851069
]
Jeff Jirsa commented on CASSANDRA-12423:
I know it's months old and not a common extension point, but it's worth
mentioning that this patch changes APIs - notably
{{CompositeType.CompositeComponent}} was public, and is now removed in a point
release. Tooling that relied on it (albeit rare) is potentially broken. It's
not a common extension point, but it was there from 3.0.0 to 3.0.8, and
removing it in 3.0.9 is unfortunate.
> Cells missing from compact storage table after upgrading from 2.1.9 to 3.7
> --
>
> Key: CASSANDRA-12423
> URL: https://issues.apache.org/jira/browse/CASSANDRA-12423
> Project: Cassandra
> Issue Type: Bug
> Components: Local Write-Read Paths
>Reporter: Tomasz Grabiec
>Assignee: Stefania
> Fix For: 3.0.9, 3.10
>
> Attachments: 12423.tar.gz
>
>
> Schema:
> {code}
> create table ks1.test ( id int, c1 text, c2 text, v int, primary key (id, c1,
> c2)) with compact storage and compression = {'sstable_compression': ''};
> {code}
> sstable2json before upgrading:
> {code}
> [
> {"key": "1",
> "cells": [["","0",1470761440040513],
>["a","asd",2470761440040513,"t",1470764842],
>["asd:","0",1470761451368658],
>["asd:asd","0",1470761449416613]]}
> ]
> {code}
> Query result with 2.1.9:
> {code}
> cqlsh> select * from ks1.test;
> id | c1 | c2 | v
> +-+--+---
> 1 | | null | 0
> 1 | asd | | 0
> 1 | asd | asd | 0
> (3 rows)
> {code}
> Query result with 3.7:
> {code}
> cqlsh> select * from ks1.test;
> id | 6331 | 6332 | v
> +--+--+---
> 1 | | null | 0
> (1 rows)
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Commented] (CASSANDRA-12423) Cells missing from compact storage table after upgrading from 2.1.9 to 3.7
[
https://issues.apache.org/jira/browse/CASSANDRA-12423?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15483567#comment-15483567
]
Stefania commented on CASSANDRA-12423:
--
The latest run of dtests on trunk was good: the only failure was a cqlsh
failure that occasionally occurs on the unpatched version of trunk as well.
Committed to 3.0 as d600f51ee1a3eb7b30ce3c409129567b70c22012 and merged into
trunk.
> Cells missing from compact storage table after upgrading from 2.1.9 to 3.7
> --
>
> Key: CASSANDRA-12423
> URL: https://issues.apache.org/jira/browse/CASSANDRA-12423
> Project: Cassandra
> Issue Type: Bug
>Reporter: Tomasz Grabiec
>Assignee: Stefania
> Attachments: 12423.tar.gz
>
>
> Schema:
> {code}
> create table ks1.test ( id int, c1 text, c2 text, v int, primary key (id, c1,
> c2)) with compact storage and compression = {'sstable_compression': ''};
> {code}
> sstable2json before upgrading:
> {code}
> [
> {"key": "1",
> "cells": [["","0",1470761440040513],
>["a","asd",2470761440040513,"t",1470764842],
>["asd:","0",1470761451368658],
>["asd:asd","0",1470761449416613]]}
> ]
> {code}
> Query result with 2.1.9:
> {code}
> cqlsh> select * from ks1.test;
> id | c1 | c2 | v
> +-+--+---
> 1 | | null | 0
> 1 | asd | | 0
> 1 | asd | asd | 0
> (3 rows)
> {code}
> Query result with 3.7:
> {code}
> cqlsh> select * from ks1.test;
> id | 6331 | 6332 | v
> +--+--+---
> 1 | | null | 0
> (1 rows)
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (CASSANDRA-12423) Cells missing from compact storage table after upgrading from 2.1.9 to 3.7
[
https://issues.apache.org/jira/browse/CASSANDRA-12423?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15482685#comment-15482685
]
Stefania commented on CASSANDRA-12423:
--
utests for 3.0 are OK now: 1 timeout and 1 known failure. dtest for 3.0
completed without failures but on trunk the build is still timing out on {{tar
czvf jenkins-stef1927-12423-b-dtest-7_logs.tar.gz \-\-ignore-failed-read logs/
/mnt/tmp/dtest-*/*/node*/logs}}. I've recreated the branch to pick up the
latest fixes for the CI build and relaunched.
> Cells missing from compact storage table after upgrading from 2.1.9 to 3.7
> --
>
> Key: CASSANDRA-12423
> URL: https://issues.apache.org/jira/browse/CASSANDRA-12423
> Project: Cassandra
> Issue Type: Bug
>Reporter: Tomasz Grabiec
>Assignee: Stefania
> Attachments: 12423.tar.gz
>
>
> Schema:
> {code}
> create table ks1.test ( id int, c1 text, c2 text, v int, primary key (id, c1,
> c2)) with compact storage and compression = {'sstable_compression': ''};
> {code}
> sstable2json before upgrading:
> {code}
> [
> {"key": "1",
> "cells": [["","0",1470761440040513],
>["a","asd",2470761440040513,"t",1470764842],
>["asd:","0",1470761451368658],
>["asd:asd","0",1470761449416613]]}
> ]
> {code}
> Query result with 2.1.9:
> {code}
> cqlsh> select * from ks1.test;
> id | c1 | c2 | v
> +-+--+---
> 1 | | null | 0
> 1 | asd | | 0
> 1 | asd | asd | 0
> (3 rows)
> {code}
> Query result with 3.7:
> {code}
> cqlsh> select * from ks1.test;
> id | 6331 | 6332 | v
> +--+--+---
> 1 | | null | 0
> (1 rows)
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (CASSANDRA-12423) Cells missing from compact storage table after upgrading from 2.1.9 to 3.7
[
https://issues.apache.org/jira/browse/CASSANDRA-12423?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15457835#comment-15457835
]
Stefania commented on CASSANDRA-12423:
--
Both dtest jobs were aborted (they timed out when collecting the artifacts) and
so I restarted them about an hour ago. I've just restarted the utests for 3.0
as well.
Thanks for the review and for improving the patch, I'll take care of committing
once CI is green, or post another update in case of issues.
> Cells missing from compact storage table after upgrading from 2.1.9 to 3.7
> --
>
> Key: CASSANDRA-12423
> URL: https://issues.apache.org/jira/browse/CASSANDRA-12423
> Project: Cassandra
> Issue Type: Bug
>Reporter: Tomasz Grabiec
>Assignee: Stefania
> Attachments: 12423.tar.gz
>
>
> Schema:
> {code}
> create table ks1.test ( id int, c1 text, c2 text, v int, primary key (id, c1,
> c2)) with compact storage and compression = {'sstable_compression': ''};
> {code}
> sstable2json before upgrading:
> {code}
> [
> {"key": "1",
> "cells": [["","0",1470761440040513],
>["a","asd",2470761440040513,"t",1470764842],
>["asd:","0",1470761451368658],
>["asd:asd","0",1470761449416613]]}
> ]
> {code}
> Query result with 2.1.9:
> {code}
> cqlsh> select * from ks1.test;
> id | c1 | c2 | v
> +-+--+---
> 1 | | null | 0
> 1 | asd | | 0
> 1 | asd | asd | 0
> (3 rows)
> {code}
> Query result with 3.7:
> {code}
> cqlsh> select * from ks1.test;
> id | 6331 | 6332 | v
> +--+--+---
> 1 | | null | 0
> (1 rows)
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (CASSANDRA-12423) Cells missing from compact storage table after upgrading from 2.1.9 to 3.7
[
https://issues.apache.org/jira/browse/CASSANDRA-12423?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15457807#comment-15457807
]
Sylvain Lebresne commented on CASSANDRA-12423:
--
bq. . I merely removed {{prefixValues}}, which was no longer required
Sorry for forgetting that one.
bq. Rebased, squashed, and cherry-picked for trunk.
Thanks. With that, I'm personally +1 on the patches as is once we have green
CI. On that front, the dtests are still running but the 3.0 testall run has 3
weird failures. They pretty clearly seems unrelated to this issue but it could
be nice re-running the job to make sure it's not a persistent problem with that
branch.
> Cells missing from compact storage table after upgrading from 2.1.9 to 3.7
> --
>
> Key: CASSANDRA-12423
> URL: https://issues.apache.org/jira/browse/CASSANDRA-12423
> Project: Cassandra
> Issue Type: Bug
>Reporter: Tomasz Grabiec
>Assignee: Stefania
> Attachments: 12423.tar.gz
>
>
> Schema:
> {code}
> create table ks1.test ( id int, c1 text, c2 text, v int, primary key (id, c1,
> c2)) with compact storage and compression = {'sstable_compression': ''};
> {code}
> sstable2json before upgrading:
> {code}
> [
> {"key": "1",
> "cells": [["","0",1470761440040513],
>["a","asd",2470761440040513,"t",1470764842],
>["asd:","0",1470761451368658],
>["asd:asd","0",1470761449416613]]}
> ]
> {code}
> Query result with 2.1.9:
> {code}
> cqlsh> select * from ks1.test;
> id | c1 | c2 | v
> +-+--+---
> 1 | | null | 0
> 1 | asd | | 0
> 1 | asd | asd | 0
> (3 rows)
> {code}
> Query result with 3.7:
> {code}
> cqlsh> select * from ks1.test;
> id | 6331 | 6332 | v
> +--+--+---
> 1 | | null | 0
> (1 rows)
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (CASSANDRA-12423) Cells missing from compact storage table after upgrading from 2.1.9 to 3.7
[
https://issues.apache.org/jira/browse/CASSANDRA-12423?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15457293#comment-15457293
]
Stefania commented on CASSANDRA-12423:
--
As far as I could see it is equivalent indeed and I think it's easier to read
like this. I merely removed {{prefixValues}}, which was no longer required, and
fixed a typo in my original comment. Rebased, squashed, and cherry-picked for
trunk (with conflicts):
||3.0||trunk||
|[patch|https://github.com/stef1927/cassandra/commits/12423-3.0-b]|[patch|https://github.com/stef1927/cassandra/commits/12423-b]|
|[testall|http://cassci.datastax.com/view/Dev/view/stef1927/job/stef1927-12423-3.0-b-testall/]|[testall|http://cassci.datastax.com/view/Dev/view/stef1927/job/stef1927-12423-b-testall/]|
|[dtest|http://cassci.datastax.com/view/Dev/view/stef1927/job/stef1927-12423-3.0-b-dtest/]|[dtest|http://cassci.datastax.com/view/Dev/view/stef1927/job/stef1927-12423-b-dtest/]|
> Cells missing from compact storage table after upgrading from 2.1.9 to 3.7
> --
>
> Key: CASSANDRA-12423
> URL: https://issues.apache.org/jira/browse/CASSANDRA-12423
> Project: Cassandra
> Issue Type: Bug
>Reporter: Tomasz Grabiec
>Assignee: Stefania
> Attachments: 12423.tar.gz
>
>
> Schema:
> {code}
> create table ks1.test ( id int, c1 text, c2 text, v int, primary key (id, c1,
> c2)) with compact storage and compression = {'sstable_compression': ''};
> {code}
> sstable2json before upgrading:
> {code}
> [
> {"key": "1",
> "cells": [["","0",1470761440040513],
>["a","asd",2470761440040513,"t",1470764842],
>["asd:","0",1470761451368658],
>["asd:asd","0",1470761449416613]]}
> ]
> {code}
> Query result with 2.1.9:
> {code}
> cqlsh> select * from ks1.test;
> id | c1 | c2 | v
> +-+--+---
> 1 | | null | 0
> 1 | asd | | 0
> 1 | asd | asd | 0
> (3 rows)
> {code}
> Query result with 3.7:
> {code}
> cqlsh> select * from ks1.test;
> id | 6331 | 6332 | v
> +--+--+---
> 1 | | null | 0
> (1 rows)
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (CASSANDRA-12423) Cells missing from compact storage table after upgrading from 2.1.9 to 3.7
[
https://issues.apache.org/jira/browse/CASSANDRA-12423?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15455359#comment-15455359
]
Sylvain Lebresne commented on CASSANDRA-12423:
--
The changes looks good, but the method feels a bit harder to read than it
really should so I took the liberty to reorganize it a bit below (it's
equivalent logic-wise though, hopefully at least).
I'm nitpicking the shit out of this though so let me know if you think that's
more readable or not really and we'll just go with whatever you prefer.
| [12423-3.0|https://github.com/pcmanus/cassandra/commits/12423-3.0] |
[utests|http://cassci.datastax.com/job/pcmanus-12423-3.0-testall] |
[dtests|http://cassci.datastax.com/job/pcmanus-12423-3.0-dtest] |
> Cells missing from compact storage table after upgrading from 2.1.9 to 3.7
> --
>
> Key: CASSANDRA-12423
> URL: https://issues.apache.org/jira/browse/CASSANDRA-12423
> Project: Cassandra
> Issue Type: Bug
>Reporter: Tomasz Grabiec
>Assignee: Stefania
> Attachments: 12423.tar.gz
>
>
> Schema:
> {code}
> create table ks1.test ( id int, c1 text, c2 text, v int, primary key (id, c1,
> c2)) with compact storage and compression = {'sstable_compression': ''};
> {code}
> sstable2json before upgrading:
> {code}
> [
> {"key": "1",
> "cells": [["","0",1470761440040513],
>["a","asd",2470761440040513,"t",1470764842],
>["asd:","0",1470761451368658],
>["asd:asd","0",1470761449416613]]}
> ]
> {code}
> Query result with 2.1.9:
> {code}
> cqlsh> select * from ks1.test;
> id | c1 | c2 | v
> +-+--+---
> 1 | | null | 0
> 1 | asd | | 0
> 1 | asd | asd | 0
> (3 rows)
> {code}
> Query result with 3.7:
> {code}
> cqlsh> select * from ks1.test;
> id | 6331 | 6332 | v
> +--+--+---
> 1 | | null | 0
> (1 rows)
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (CASSANDRA-12423) Cells missing from compact storage table after upgrading from 2.1.9 to 3.7
[
https://issues.apache.org/jira/browse/CASSANDRA-12423?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15454222#comment-15454222
]
Stefania commented on CASSANDRA-12423:
--
I've created a dtest pull request
[here|https://github.com/riptano/cassandra-dtest/pull/1297].
> Cells missing from compact storage table after upgrading from 2.1.9 to 3.7
> --
>
> Key: CASSANDRA-12423
> URL: https://issues.apache.org/jira/browse/CASSANDRA-12423
> Project: Cassandra
> Issue Type: Bug
>Reporter: Tomasz Grabiec
>Assignee: Stefania
> Attachments: 12423.tar.gz
>
>
> Schema:
> {code}
> create table ks1.test ( id int, c1 text, c2 text, v int, primary key (id, c1,
> c2)) with compact storage and compression = {'sstable_compression': ''};
> {code}
> sstable2json before upgrading:
> {code}
> [
> {"key": "1",
> "cells": [["","0",1470761440040513],
>["a","asd",2470761440040513,"t",1470764842],
>["asd:","0",1470761451368658],
>["asd:asd","0",1470761449416613]]}
> ]
> {code}
> Query result with 2.1.9:
> {code}
> cqlsh> select * from ks1.test;
> id | c1 | c2 | v
> +-+--+---
> 1 | | null | 0
> 1 | asd | | 0
> 1 | asd | asd | 0
> (3 rows)
> {code}
> Query result with 3.7:
> {code}
> cqlsh> select * from ks1.test;
> id | 6331 | 6332 | v
> +--+--+---
> 1 | | null | 0
> (1 rows)
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (CASSANDRA-12423) Cells missing from compact storage table after upgrading from 2.1.9 to 3.7
[
https://issues.apache.org/jira/browse/CASSANDRA-12423?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15454204#comment-15454204
]
Stefania commented on CASSANDRA-12423:
--
I've moved the logic of the patch to {{decodeBound}} and removed
{{CompositeComponent}} in the [3.0
patch|https://github.com/stef1927/cassandra/commits/12423-3.0-b], see if it
looks better now [~slebresne].
I think we should also split out a new method to decode the bound kind, I did
not do it yet because it makes comparing harder.
> Cells missing from compact storage table after upgrading from 2.1.9 to 3.7
> --
>
> Key: CASSANDRA-12423
> URL: https://issues.apache.org/jira/browse/CASSANDRA-12423
> Project: Cassandra
> Issue Type: Bug
>Reporter: Tomasz Grabiec
>Assignee: Stefania
> Attachments: 12423.tar.gz
>
>
> Schema:
> {code}
> create table ks1.test ( id int, c1 text, c2 text, v int, primary key (id, c1,
> c2)) with compact storage and compression = {'sstable_compression': ''};
> {code}
> sstable2json before upgrading:
> {code}
> [
> {"key": "1",
> "cells": [["","0",1470761440040513],
>["a","asd",2470761440040513,"t",1470764842],
>["asd:","0",1470761451368658],
>["asd:asd","0",1470761449416613]]}
> ]
> {code}
> Query result with 2.1.9:
> {code}
> cqlsh> select * from ks1.test;
> id | c1 | c2 | v
> +-+--+---
> 1 | | null | 0
> 1 | asd | | 0
> 1 | asd | asd | 0
> (3 rows)
> {code}
> Query result with 3.7:
> {code}
> cqlsh> select * from ks1.test;
> id | 6331 | 6332 | v
> +--+--+---
> 1 | | null | 0
> (1 rows)
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (CASSANDRA-12423) Cells missing from compact storage table after upgrading from 2.1.9 to 3.7
[
https://issues.apache.org/jira/browse/CASSANDRA-12423?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15451634#comment-15451634
]
Sylvain Lebresne commented on CASSANDRA-12423:
--
The patch lgtm logic-wise, but the nit I'd have is that this somewhat
specialize the {{CompositeType.deconstruct()}} method for this backward
compatibility problem (granted it's only use in {{LegacyLayout}}, but before
this patch, the method somewhat made sense as a {{CompositeType}} generic
method. not so much anymore). So I'd suggest moving the logic out of
{{deconstruct()}} (actually, we could fairly easily ditch {{deconstruct()}} and
{{CompositeComponent}} altogether since you can achieve the same results using
{{CompositeType.splitName}} and {{CompositeType.lastEOC}} easily, but I'm fine
not bothering with that).
> Cells missing from compact storage table after upgrading from 2.1.9 to 3.7
> --
>
> Key: CASSANDRA-12423
> URL: https://issues.apache.org/jira/browse/CASSANDRA-12423
> Project: Cassandra
> Issue Type: Bug
>Reporter: Tomasz Grabiec
>Assignee: Stefania
> Attachments: 12423.tar.gz
>
>
> Schema:
> {code}
> create table ks1.test ( id int, c1 text, c2 text, v int, primary key (id, c1,
> c2)) with compact storage and compression = {'sstable_compression': ''};
> {code}
> sstable2json before upgrading:
> {code}
> [
> {"key": "1",
> "cells": [["","0",1470761440040513],
>["a","asd",2470761440040513,"t",1470764842],
>["asd:","0",1470761451368658],
>["asd:asd","0",1470761449416613]]}
> ]
> {code}
> Query result with 2.1.9:
> {code}
> cqlsh> select * from ks1.test;
> id | c1 | c2 | v
> +-+--+---
> 1 | | null | 0
> 1 | asd | | 0
> 1 | asd | asd | 0
> (3 rows)
> {code}
> Query result with 3.7:
> {code}
> cqlsh> select * from ks1.test;
> id | 6331 | 6332 | v
> +--+--+---
> 1 | | null | 0
> (1 rows)
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (CASSANDRA-12423) Cells missing from compact storage table after upgrading from 2.1.9 to 3.7
[
https://issues.apache.org/jira/browse/CASSANDRA-12423?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15451523#comment-15451523
]
Stefania commented on CASSANDRA-12423:
--
CI for 3.0 completed with no failures, so from my analysis above CI on trunk
should be fine as well, but I can relaunch it on trunk if there are concerns.
> Cells missing from compact storage table after upgrading from 2.1.9 to 3.7
> --
>
> Key: CASSANDRA-12423
> URL: https://issues.apache.org/jira/browse/CASSANDRA-12423
> Project: Cassandra
> Issue Type: Bug
>Reporter: Tomasz Grabiec
>Assignee: Stefania
> Attachments: 12423.tar.gz
>
>
> Schema:
> {code}
> create table ks1.test ( id int, c1 text, c2 text, v int, primary key (id, c1,
> c2)) with compact storage and compression = {'sstable_compression': ''};
> {code}
> sstable2json before upgrading:
> {code}
> [
> {"key": "1",
> "cells": [["","0",1470761440040513],
>["a","asd",2470761440040513,"t",1470764842],
>["asd:","0",1470761451368658],
>["asd:asd","0",1470761449416613]]}
> ]
> {code}
> Query result with 2.1.9:
> {code}
> cqlsh> select * from ks1.test;
> id | c1 | c2 | v
> +-+--+---
> 1 | | null | 0
> 1 | asd | | 0
> 1 | asd | asd | 0
> (3 rows)
> {code}
> Query result with 3.7:
> {code}
> cqlsh> select * from ks1.test;
> id | 6331 | 6332 | v
> +--+--+---
> 1 | | null | 0
> (1 rows)
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (CASSANDRA-12423) Cells missing from compact storage table after upgrading from 2.1.9 to 3.7
[
https://issues.apache.org/jira/browse/CASSANDRA-12423?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15451011#comment-15451011
]
Stefania commented on CASSANDRA-12423:
--
Thanks for restarting the dtests.
There is one failing dtest for both branches, it is a new dtest that was added
yesterday. It passed locally but only after rebasing. There is also a utest
failing for 3.0, that too passes locally. The utest failing for trunk is
instead a known issue that fails on unpatched trunk too.
I've relaunched both utests and dtests but only for 3.0.
> Cells missing from compact storage table after upgrading from 2.1.9 to 3.7
> --
>
> Key: CASSANDRA-12423
> URL: https://issues.apache.org/jira/browse/CASSANDRA-12423
> Project: Cassandra
> Issue Type: Bug
>Reporter: Tomasz Grabiec
>Assignee: Stefania
> Attachments: 12423.tar.gz
>
>
> Schema:
> {code}
> create table ks1.test ( id int, c1 text, c2 text, v int, primary key (id, c1,
> c2)) with compact storage and compression = {'sstable_compression': ''};
> {code}
> sstable2json before upgrading:
> {code}
> [
> {"key": "1",
> "cells": [["","0",1470761440040513],
>["a","asd",2470761440040513,"t",1470764842],
>["asd:","0",1470761451368658],
>["asd:asd","0",1470761449416613]]}
> ]
> {code}
> Query result with 2.1.9:
> {code}
> cqlsh> select * from ks1.test;
> id | c1 | c2 | v
> +-+--+---
> 1 | | null | 0
> 1 | asd | | 0
> 1 | asd | asd | 0
> (3 rows)
> {code}
> Query result with 3.7:
> {code}
> cqlsh> select * from ks1.test;
> id | 6331 | 6332 | v
> +--+--+---
> 1 | | null | 0
> (1 rows)
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (CASSANDRA-12423) Cells missing from compact storage table after upgrading from 2.1.9 to 3.7
[
https://issues.apache.org/jira/browse/CASSANDRA-12423?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15449539#comment-15449539
]
Tyler Hobbs commented on CASSANDRA-12423:
-
Using an extra empty component is definitely a clever idea. Nice job,
[~slebresne].
It looks like the dtests are having some problems at the test-runner level, so
I've restarted both of them.
> Cells missing from compact storage table after upgrading from 2.1.9 to 3.7
> --
>
> Key: CASSANDRA-12423
> URL: https://issues.apache.org/jira/browse/CASSANDRA-12423
> Project: Cassandra
> Issue Type: Bug
>Reporter: Tomasz Grabiec
>Assignee: Stefania
> Attachments: 12423.tar.gz
>
>
> Schema:
> {code}
> create table ks1.test ( id int, c1 text, c2 text, v int, primary key (id, c1,
> c2)) with compact storage and compression = {'sstable_compression': ''};
> {code}
> sstable2json before upgrading:
> {code}
> [
> {"key": "1",
> "cells": [["","0",1470761440040513],
>["a","asd",2470761440040513,"t",1470764842],
>["asd:","0",1470761451368658],
>["asd:asd","0",1470761449416613]]}
> ]
> {code}
> Query result with 2.1.9:
> {code}
> cqlsh> select * from ks1.test;
> id | c1 | c2 | v
> +-+--+---
> 1 | | null | 0
> 1 | asd | | 0
> 1 | asd | asd | 0
> (3 rows)
> {code}
> Query result with 3.7:
> {code}
> cqlsh> select * from ks1.test;
> id | 6331 | 6332 | v
> +--+--+---
> 1 | | null | 0
> (1 rows)
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (CASSANDRA-12423) Cells missing from compact storage table after upgrading from 2.1.9 to 3.7
[
https://issues.apache.org/jira/browse/CASSANDRA-12423?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15448448#comment-15448448
]
Stefania commented on CASSANDRA-12423:
--
This is awesome, I had no idea we could achieve the same behavior by adding an
empty component and changing the bound type, thank you so much [~slebresne]!
Here is the new patch, CI still pending:
||3.0||trunk||
|[patch|https://github.com/stef1927/cassandra/commits/12423-3.0-b]|[patch|https://github.com/stef1927/cassandra/commits/12423-b]|
|[testall|http://cassci.datastax.com/view/Dev/view/stef1927/job/stef1927-12423-3.0-b-testall/]|[testall|http://cassci.datastax.com/view/Dev/view/stef1927/job/stef1927-12423-b-testall/]|
|[dtest|http://cassci.datastax.com/view/Dev/view/stef1927/job/stef1927-12423-3.0-b-dtest/]|[dtest|http://cassci.datastax.com/view/Dev/view/stef1927/job/stef1927-12423-b-dtest/]|
> Cells missing from compact storage table after upgrading from 2.1.9 to 3.7
> --
>
> Key: CASSANDRA-12423
> URL: https://issues.apache.org/jira/browse/CASSANDRA-12423
> Project: Cassandra
> Issue Type: Bug
>Reporter: Tomasz Grabiec
>Assignee: Stefania
> Attachments: 12423.tar.gz
>
>
> Schema:
> {code}
> create table ks1.test ( id int, c1 text, c2 text, v int, primary key (id, c1,
> c2)) with compact storage and compression = {'sstable_compression': ''};
> {code}
> sstable2json before upgrading:
> {code}
> [
> {"key": "1",
> "cells": [["","0",1470761440040513],
>["a","asd",2470761440040513,"t",1470764842],
>["asd:","0",1470761451368658],
>["asd:asd","0",1470761449416613]]}
> ]
> {code}
> Query result with 2.1.9:
> {code}
> cqlsh> select * from ks1.test;
> id | c1 | c2 | v
> +-+--+---
> 1 | | null | 0
> 1 | asd | | 0
> 1 | asd | asd | 0
> (3 rows)
> {code}
> Query result with 3.7:
> {code}
> cqlsh> select * from ks1.test;
> id | 6331 | 6332 | v
> +--+--+---
> 1 | | null | 0
> (1 rows)
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (CASSANDRA-12423) Cells missing from compact storage table after upgrading from 2.1.9 to 3.7
[
https://issues.apache.org/jira/browse/CASSANDRA-12423?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15445679#comment-15445679
]
Sylvain Lebresne commented on CASSANDRA-12423:
--
I'm not sure about the solution of this patch. First, I'm bothered with
"polluting" the storage engine just for this; I fear we'll have to drag that on
for a long time if we do so and that's annoying. Second, I think this imply
really complex upgrade instructions and not a very good user experience.
And I believe there is a simpler fix.
The problem is when a end bound is composite, has only a prefix of all the
components and the EOC is 0. In that case, it means the tombstone (or slice)
goes up to the prefix, including it, but not further. In other words, anything
with that prefix but other component is excluded. We can get that easily by
making an exlusive end until the prefix *plus* a empty component after it.
To take an example, say I use the syntax {{\[0\]}} to denote a 2
component composite with a 0 EOC. The end range tombstone of Tomasz above is
{{<'asd'>\[0\]}}, but it would be equivalent to use {{<'asd'><>\[-1\]}} instead
(that is, adding an empty component as 2nd component instead of not having one
at all, but excluding that exact "full" clustering). It's equivalent because
{{<'asd'><>}} is the very next item ordering after {{<'asd'>}}, and so going up
until the latter one inclusive is the same than going up to the former
exclusive.
And we can easily represent that {{<'asd'><>\[-1\]}} composite in 3.0 (it just
uses {{EXCL_END_BOUND}}), so I think that's what we should return from
{{LegacyLayout.decodeBound()}} in that case.
> Cells missing from compact storage table after upgrading from 2.1.9 to 3.7
> --
>
> Key: CASSANDRA-12423
> URL: https://issues.apache.org/jira/browse/CASSANDRA-12423
> Project: Cassandra
> Issue Type: Bug
>Reporter: Tomasz Grabiec
>Assignee: Stefania
> Attachments: 12423.tar.gz
>
>
> Schema:
> {code}
> create table ks1.test ( id int, c1 text, c2 text, v int, primary key (id, c1,
> c2)) with compact storage and compression = {'sstable_compression': ''};
> {code}
> sstable2json before upgrading:
> {code}
> [
> {"key": "1",
> "cells": [["","0",1470761440040513],
>["a","asd",2470761440040513,"t",1470764842],
>["asd:","0",1470761451368658],
>["asd:asd","0",1470761449416613]]}
> ]
> {code}
> Query result with 2.1.9:
> {code}
> cqlsh> select * from ks1.test;
> id | c1 | c2 | v
> +-+--+---
> 1 | | null | 0
> 1 | asd | | 0
> 1 | asd | asd | 0
> (3 rows)
> {code}
> Query result with 3.7:
> {code}
> cqlsh> select * from ks1.test;
> id | 6331 | 6332 | v
> +--+--+---
> 1 | | null | 0
> (1 rows)
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (CASSANDRA-12423) Cells missing from compact storage table after upgrading from 2.1.9 to 3.7
[
https://issues.apache.org/jira/browse/CASSANDRA-12423?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15438394#comment-15438394
]
Stefania commented on CASSANDRA-12423:
--
I agree that {{INCL_END_BOUND}} is much safer than {{EXCL_END_BOUND}} but my
problem was not so much the tests (they can be fixed by simply adding the flag)
but that now Thrift users may end up with slice queries missing the final cell
unless they enable the flag.
Anyway, I think you are correct in that it is too dangerous to leave
{{EXCL_END_BOUND}} and so I've amended the latest commit and tried to explain
best I could in NEWS.txt. Repeating the links here for convenience:
||3.0||trunk||
|[patch|https://github.com/stef1927/cassandra/commits/12423-3.0]|[patch|https://github.com/stef1927/cassandra/commits/12423]|
|[testall|http://cassci.datastax.com/view/Dev/view/stef1927/job/stef1927-12423-3.0-testall/]|[testall|http://cassci.datastax.com/view/Dev/view/stef1927/job/stef1927-12423-testall/]|
|[dtest|http://cassci.datastax.com/view/Dev/view/stef1927/job/stef1927-12423-3.0-dtest/]|[dtest|http://cassci.datastax.com/view/Dev/view/stef1927/job/stef1927-12423-dtest/]|
CI pending.
> Cells missing from compact storage table after upgrading from 2.1.9 to 3.7
> --
>
> Key: CASSANDRA-12423
> URL: https://issues.apache.org/jira/browse/CASSANDRA-12423
> Project: Cassandra
> Issue Type: Bug
>Reporter: Tomasz Grabiec
>Assignee: Stefania
> Attachments: 12423.tar.gz
>
>
> Schema:
> {code}
> create table ks1.test ( id int, c1 text, c2 text, v int, primary key (id, c1,
> c2)) with compact storage and compression = {'sstable_compression': ''};
> {code}
> sstable2json before upgrading:
> {code}
> [
> {"key": "1",
> "cells": [["","0",1470761440040513],
>["a","asd",2470761440040513,"t",1470764842],
>["asd:","0",1470761451368658],
>["asd:asd","0",1470761449416613]]}
> ]
> {code}
> Query result with 2.1.9:
> {code}
> cqlsh> select * from ks1.test;
> id | c1 | c2 | v
> +-+--+---
> 1 | | null | 0
> 1 | asd | | 0
> 1 | asd | asd | 0
> (3 rows)
> {code}
> Query result with 3.7:
> {code}
> cqlsh> select * from ks1.test;
> id | 6331 | 6332 | v
> +--+--+---
> 1 | | null | 0
> (1 rows)
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (CASSANDRA-12423) Cells missing from compact storage table after upgrading from 2.1.9 to 3.7
[
https://issues.apache.org/jira/browse/CASSANDRA-12423?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15437418#comment-15437418
]
Tyler Hobbs commented on CASSANDRA-12423:
-
bq. Just to avoid confusion, this should be committed in both 3.0.x and 3.x,
right?
Yes.
bq. Using EXCL_END_BOUND breaks the existing thrift slice query tests because
they expect an inclusive end bound with eoc=0, but I've applied the same idea
with INCL_END_BOUND instead
Hmm, I think we may want to change the Thrift tests instead. The reason that I
prefer {{EXCL_END_BOUND}} is that it can affect at most one cell, and that cell
will be incorrectly left alive (which means it can be deleted later, if
necessary). In contrast, {{INCL_END_BOUND}} can erroneously delete an
unbounded number of cells, and there is no operator recourse for this.
> Cells missing from compact storage table after upgrading from 2.1.9 to 3.7
> --
>
> Key: CASSANDRA-12423
> URL: https://issues.apache.org/jira/browse/CASSANDRA-12423
> Project: Cassandra
> Issue Type: Bug
>Reporter: Tomasz Grabiec
>Assignee: Stefania
> Attachments: 12423.tar.gz
>
>
> Schema:
> {code}
> create table ks1.test ( id int, c1 text, c2 text, v int, primary key (id, c1,
> c2)) with compact storage and compression = {'sstable_compression': ''};
> {code}
> sstable2json before upgrading:
> {code}
> [
> {"key": "1",
> "cells": [["","0",1470761440040513],
>["a","asd",2470761440040513,"t",1470764842],
>["asd:","0",1470761451368658],
>["asd:asd","0",1470761449416613]]}
> ]
> {code}
> Query result with 2.1.9:
> {code}
> cqlsh> select * from ks1.test;
> id | c1 | c2 | v
> +-+--+---
> 1 | | null | 0
> 1 | asd | | 0
> 1 | asd | asd | 0
> (3 rows)
> {code}
> Query result with 3.7:
> {code}
> cqlsh> select * from ks1.test;
> id | 6331 | 6332 | v
> +--+--+---
> 1 | | null | 0
> (1 rows)
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (CASSANDRA-12423) Cells missing from compact storage table after upgrading from 2.1.9 to 3.7
[
https://issues.apache.org/jira/browse/CASSANDRA-12423?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15436588#comment-15436588
]
Stefania commented on CASSANDRA-12423:
--
Using {{EXCL_END_BOUND}} breaks the existing thrift slice query tests because
they expect an inclusive end bound with eoc=0, but I've applied the same idea
with {{INCL_END_BOUND}} instead. See the [latest
commit|https://github.com/stef1927/cassandra/commit/50964cc8fcce94911ddc057f111445e5b95618dd].
I've also restarted the CI tests.
Just to avoid confusion, this should be committed in both 3.0.x and 3.x, right?
> Cells missing from compact storage table after upgrading from 2.1.9 to 3.7
> --
>
> Key: CASSANDRA-12423
> URL: https://issues.apache.org/jira/browse/CASSANDRA-12423
> Project: Cassandra
> Issue Type: Bug
>Reporter: Tomasz Grabiec
>Assignee: Stefania
> Attachments: 12423.tar.gz
>
>
> Schema:
> {code}
> create table ks1.test ( id int, c1 text, c2 text, v int, primary key (id, c1,
> c2)) with compact storage and compression = {'sstable_compression': ''};
> {code}
> sstable2json before upgrading:
> {code}
> [
> {"key": "1",
> "cells": [["","0",1470761440040513],
>["a","asd",2470761440040513,"t",1470764842],
>["asd:","0",1470761451368658],
>["asd:asd","0",1470761449416613]]}
> ]
> {code}
> Query result with 2.1.9:
> {code}
> cqlsh> select * from ks1.test;
> id | c1 | c2 | v
> +-+--+---
> 1 | | null | 0
> 1 | asd | | 0
> 1 | asd | asd | 0
> (3 rows)
> {code}
> Query result with 3.7:
> {code}
> cqlsh> select * from ks1.test;
> id | 6331 | 6332 | v
> +--+--+---
> 1 | | null | 0
> (1 rows)
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (CASSANDRA-12423) Cells missing from compact storage table after upgrading from 2.1.9 to 3.7
[
https://issues.apache.org/jira/browse/CASSANDRA-12423?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15435201#comment-15435201
]
Tyler Hobbs commented on CASSANDRA-12423:
-
I think the idea of a flag is good. How about something like this: without the
flag enabled, we treat eoc=0 bounds as {{EXCL_END_BOUND}} (since that's the
closest to the correct behavior) and log a warning that explains what's going
on and mentions the flag. The NEWS.txt entry could also suggest disabling
autocompaction until the flag has been enabled to avoid permanently converting
those bounds to something slightly incorrect.
That's not an amazing solution, but I do agree that this case should be very
rare in the real world.
> Cells missing from compact storage table after upgrading from 2.1.9 to 3.7
> --
>
> Key: CASSANDRA-12423
> URL: https://issues.apache.org/jira/browse/CASSANDRA-12423
> Project: Cassandra
> Issue Type: Bug
>Reporter: Tomasz Grabiec
>Assignee: Stefania
> Attachments: 12423.tar.gz
>
>
> Schema:
> {code}
> create table ks1.test ( id int, c1 text, c2 text, v int, primary key (id, c1,
> c2)) with compact storage and compression = {'sstable_compression': ''};
> {code}
> sstable2json before upgrading:
> {code}
> [
> {"key": "1",
> "cells": [["","0",1470761440040513],
>["a","asd",2470761440040513,"t",1470764842],
>["asd:","0",1470761451368658],
>["asd:asd","0",1470761449416613]]}
> ]
> {code}
> Query result with 2.1.9:
> {code}
> cqlsh> select * from ks1.test;
> id | c1 | c2 | v
> +-+--+---
> 1 | | null | 0
> 1 | asd | | 0
> 1 | asd | asd | 0
> (3 rows)
> {code}
> Query result with 3.7:
> {code}
> cqlsh> select * from ks1.test;
> id | 6331 | 6332 | v
> +--+--+---
> 1 | | null | 0
> (1 rows)
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (CASSANDRA-12423) Cells missing from compact storage table after upgrading from 2.1.9 to 3.7
[
https://issues.apache.org/jira/browse/CASSANDRA-12423?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15434242#comment-15434242
]
Stefania commented on CASSANDRA-12423:
--
Thanks for the review!
I've renamed {{INCL_END_BOUND_EOC_0}} to {{END_BOUND_EOC_0}} and I've updated
the comment as you suggested. I appreciate your explanation and your test as
the exact behavior wasn't 100% clear to me as well.
I've picked up your test in [this
branch|https://github.com/stef1927/cassandra-dtest/commits/12423] and I will
create a pull request shortly after resolving this ticket.
In terms of mixed-version clusters, I don't think we can risk deserialization
exceptions and the second option is not ideal, plus it's risky to trust people
to do the right thing. What we want is to ideally detect a rolling upgrade with
other 3.0.x or 3.x hosts without the fix and disable this fix until the rolling
upgrade is completed. If range tombstones with EOC=0 are uncommon amongst the
majority of users, we could simply put the fix behind a flag and tell people
who have EOC=0 range tombstones to enable the flag after the rolling upgrade is
complete but before upgrading sstables. This would make it safer to deploy this
in 3.0.x as well as 3.x. The price is inconvenience for the users affected, but
given this is a thrift issue, I would expect those users to be experienced
enough to cope, provided we document it well in NEWS.TXT. What do you think?
> Cells missing from compact storage table after upgrading from 2.1.9 to 3.7
> --
>
> Key: CASSANDRA-12423
> URL: https://issues.apache.org/jira/browse/CASSANDRA-12423
> Project: Cassandra
> Issue Type: Bug
>Reporter: Tomasz Grabiec
>Assignee: Stefania
> Attachments: 12423.tar.gz
>
>
> Schema:
> {code}
> create table ks1.test ( id int, c1 text, c2 text, v int, primary key (id, c1,
> c2)) with compact storage and compression = {'sstable_compression': ''};
> {code}
> sstable2json before upgrading:
> {code}
> [
> {"key": "1",
> "cells": [["","0",1470761440040513],
>["a","asd",2470761440040513,"t",1470764842],
>["asd:","0",1470761451368658],
>["asd:asd","0",1470761449416613]]}
> ]
> {code}
> Query result with 2.1.9:
> {code}
> cqlsh> select * from ks1.test;
> id | c1 | c2 | v
> +-+--+---
> 1 | | null | 0
> 1 | asd | | 0
> 1 | asd | asd | 0
> (3 rows)
> {code}
> Query result with 3.7:
> {code}
> cqlsh> select * from ks1.test;
> id | 6331 | 6332 | v
> +--+--+---
> 1 | | null | 0
> (1 rows)
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (CASSANDRA-12423) Cells missing from compact storage table after upgrading from 2.1.9 to 3.7
[
https://issues.apache.org/jira/browse/CASSANDRA-12423?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15433801#comment-15433801
]
Tyler Hobbs commented on CASSANDRA-12423:
-
Regarding the patch, it took me a while to convince myself this was the correct
behavior, so I'll write out some comments for posterity.
In the 2.x world, a range tombstone with a final EOC of 0 results in a
exclusive deletion _except_ for cells that exactly match the tombstone bound.
I made a
[dtest|https://github.com/thobbs/cassandra-dtest/commit/e76250a4471a81a1ece93cc954f42d4a61aa8c2d]
to double check this (which we should commit, I guess). This is why we can't
just use {{EXCL_END_BOUND}} in 3.0 -- in the case where the range tombstone
bound exactly matches a {{ClusteringPrefix}}, we want that to be included in
the deletion.
I think we should change the new enum name from {{INCL_END_BOUND_EOC_0}} to
{{END_BOUND_EOC_0}} for clarity. It's sort of both inclusive and exclusive, so
having either one in the name is confusing, I think. Regarding the code
comment:
bq. This value compares as an inclusive end bound but before any clustering
values. It can be removed once support for thrift and legacy sstables is
removed.
I find the first half of that confusing. Maybe rephrase it as "This value is
inclusive of any exact matches, but does not include any clusterings that it is
merely a prefix of." I believe the second half of the comment is incorrect, as
we'll need to continue treating those range tombstones correctly as long as
they exist. Furthermore, once we've serialized tombstones with that enum
value, we'll always need it for deserialization.
My last concern is around compatibility in a mixed-version cluster. It seems
like there could be problems if you did something like this:
* Have a 2.x cluster with range tombstones with an eoc=0 end bound
* Upgrade to some 3.x that doesn't include this fix
* Then, upgrade some nodes to a 3.y that _does_ include this fix while there
are still legacy sstables
It seems like the 3.y nodes could send results with the new enum value to the
3.x nodes which would then error while deserializing the results. I'm not sure
what to do about that yet. We could require all sstables to be upgraded before
upgrading to 3.10 from 3.x, but that would also force any remaining legacy
tombstones like this to be permanently treated incorrectly, which kind of sucks.
> Cells missing from compact storage table after upgrading from 2.1.9 to 3.7
> --
>
> Key: CASSANDRA-12423
> URL: https://issues.apache.org/jira/browse/CASSANDRA-12423
> Project: Cassandra
> Issue Type: Bug
>Reporter: Tomasz Grabiec
>Assignee: Stefania
> Attachments: 12423.tar.gz
>
>
> Schema:
> {code}
> create table ks1.test ( id int, c1 text, c2 text, v int, primary key (id, c1,
> c2)) with compact storage and compression = {'sstable_compression': ''};
> {code}
> sstable2json before upgrading:
> {code}
> [
> {"key": "1",
> "cells": [["","0",1470761440040513],
>["a","asd",2470761440040513,"t",1470764842],
>["asd:","0",1470761451368658],
>["asd:asd","0",1470761449416613]]}
> ]
> {code}
> Query result with 2.1.9:
> {code}
> cqlsh> select * from ks1.test;
> id | c1 | c2 | v
> +-+--+---
> 1 | | null | 0
> 1 | asd | | 0
> 1 | asd | asd | 0
> (3 rows)
> {code}
> Query result with 3.7:
> {code}
> cqlsh> select * from ks1.test;
> id | 6331 | 6332 | v
> +--+--+---
> 1 | | null | 0
> (1 rows)
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (CASSANDRA-12423) Cells missing from compact storage table after upgrading from 2.1.9 to 3.7
[
https://issues.apache.org/jira/browse/CASSANDRA-12423?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15430959#comment-15430959
]
Adam Holmberg commented on CASSANDRA-12423:
---
We have [PYTHON-213|https://datastax-oss.atlassian.net/browse/PYTHON-213] open.
I believe I addressed what we discussed to the extent possible. There is a gist
linked there for your review if you're willing to take a look.
> Cells missing from compact storage table after upgrading from 2.1.9 to 3.7
> --
>
> Key: CASSANDRA-12423
> URL: https://issues.apache.org/jira/browse/CASSANDRA-12423
> Project: Cassandra
> Issue Type: Bug
>Reporter: Tomasz Grabiec
>Assignee: Stefania
> Attachments: 12423.tar.gz
>
>
> Schema:
> {code}
> create table ks1.test ( id int, c1 text, c2 text, v int, primary key (id, c1,
> c2)) with compact storage and compression = {'sstable_compression': ''};
> {code}
> sstable2json before upgrading:
> {code}
> [
> {"key": "1",
> "cells": [["","0",1470761440040513],
>["a","asd",2470761440040513,"t",1470764842],
>["asd:","0",1470761451368658],
>["asd:asd","0",1470761449416613]]}
> ]
> {code}
> Query result with 2.1.9:
> {code}
> cqlsh> select * from ks1.test;
> id | c1 | c2 | v
> +-+--+---
> 1 | | null | 0
> 1 | asd | | 0
> 1 | asd | asd | 0
> (3 rows)
> {code}
> Query result with 3.7:
> {code}
> cqlsh> select * from ks1.test;
> id | 6331 | 6332 | v
> +--+--+---
> 1 | | null | 0
> (1 rows)
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (CASSANDRA-12423) Cells missing from compact storage table after upgrading from 2.1.9 to 3.7
[
https://issues.apache.org/jira/browse/CASSANDRA-12423?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15430784#comment-15430784
]
Aleksey Yeschenko commented on CASSANDRA-12423:
---
Looked into it more, together with Stefania. To sum it up, in no particular
order:
1. {{DESCRIBE}} output is broken. Sparse compact tables (with regular columns
defined) that have non-text comparators should be filtered out from
{{DESCRIBE}} output, as they cannot currently be represented via CQL {{CREATE
TABLE}}. [~aholmber] I've raised this issue privately, do you remember if there
is a python-driver JIRA open for it already? (also for filtering out supers).
[~ifesdjeen] same goes for a follow-up JIRA to CASSANDRA-7190
2. The column names for {{Keyspace1.Standard3}} in schema tables in 2.1 and 3.0
are both correct. Should be using {{BytesType}} for regulars in 2.1 (statics in
3.0) given the {{BytesType}} comparator.
3. Yup, you are pretty much right about {{ColumnIdentifier.getInterned()}}
being relied on incorrectly. We should fix this in 2.1 and upwards. I'll open a
separate JIRA (assign to you, if you don't mind it), and link to this one, to
either make the type part of the key in the interned map, or just limit its use.
> Cells missing from compact storage table after upgrading from 2.1.9 to 3.7
> --
>
> Key: CASSANDRA-12423
> URL: https://issues.apache.org/jira/browse/CASSANDRA-12423
> Project: Cassandra
> Issue Type: Bug
>Reporter: Tomasz Grabiec
>Assignee: Stefania
> Attachments: 12423.tar.gz
>
>
> Schema:
> {code}
> create table ks1.test ( id int, c1 text, c2 text, v int, primary key (id, c1,
> c2)) with compact storage and compression = {'sstable_compression': ''};
> {code}
> sstable2json before upgrading:
> {code}
> [
> {"key": "1",
> "cells": [["","0",1470761440040513],
>["a","asd",2470761440040513,"t",1470764842],
>["asd:","0",1470761451368658],
>["asd:asd","0",1470761449416613]]}
> ]
> {code}
> Query result with 2.1.9:
> {code}
> cqlsh> select * from ks1.test;
> id | c1 | c2 | v
> +-+--+---
> 1 | | null | 0
> 1 | asd | | 0
> 1 | asd | asd | 0
> (3 rows)
> {code}
> Query result with 3.7:
> {code}
> cqlsh> select * from ks1.test;
> id | 6331 | 6332 | v
> +--+--+---
> 1 | | null | 0
> (1 rows)
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (CASSANDRA-12423) Cells missing from compact storage table after upgrading from 2.1.9 to 3.7
[
https://issues.apache.org/jira/browse/CASSANDRA-12423?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15430720#comment-15430720
]
Aleksey Yeschenko commented on CASSANDRA-12423:
---
As for {{ColumnIdentifier.getInterned()}} use, I still haven't looked at the
exact issue you are facing here, so it may or may not be related. *But* overall
this is definitely a potential problem. It's been broken since forever, and
should be fixed in 2.1 upwards. The bytebuffer name does not unambiguously
represent a column identifier, the type should be part of the key.
> Cells missing from compact storage table after upgrading from 2.1.9 to 3.7
> --
>
> Key: CASSANDRA-12423
> URL: https://issues.apache.org/jira/browse/CASSANDRA-12423
> Project: Cassandra
> Issue Type: Bug
>Reporter: Tomasz Grabiec
>Assignee: Stefania
> Attachments: 12423.tar.gz
>
>
> Schema:
> {code}
> create table ks1.test ( id int, c1 text, c2 text, v int, primary key (id, c1,
> c2)) with compact storage and compression = {'sstable_compression': ''};
> {code}
> sstable2json before upgrading:
> {code}
> [
> {"key": "1",
> "cells": [["","0",1470761440040513],
>["a","asd",2470761440040513,"t",1470764842],
>["asd:","0",1470761451368658],
>["asd:asd","0",1470761449416613]]}
> ]
> {code}
> Query result with 2.1.9:
> {code}
> cqlsh> select * from ks1.test;
> id | c1 | c2 | v
> +-+--+---
> 1 | | null | 0
> 1 | asd | | 0
> 1 | asd | asd | 0
> (3 rows)
> {code}
> Query result with 3.7:
> {code}
> cqlsh> select * from ks1.test;
> id | 6331 | 6332 | v
> +--+--+---
> 1 | | null | 0
> (1 rows)
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (CASSANDRA-12423) Cells missing from compact storage table after upgrading from 2.1.9 to 3.7
[
https://issues.apache.org/jira/browse/CASSANDRA-12423?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15430701#comment-15430701
]
Aleksey Yeschenko commented on CASSANDRA-12423:
---
Looking into it. That said, there have been some changes to
{{CompactTables.columnDefinitionComparator()}} method recently in 3.0/trunk.
Only {{regular}} column names should use the comparator to decode/encode the
column name, never clustering columns.
> Cells missing from compact storage table after upgrading from 2.1.9 to 3.7
> --
>
> Key: CASSANDRA-12423
> URL: https://issues.apache.org/jira/browse/CASSANDRA-12423
> Project: Cassandra
> Issue Type: Bug
>Reporter: Tomasz Grabiec
>Assignee: Stefania
> Attachments: 12423.tar.gz
>
>
> Schema:
> {code}
> create table ks1.test ( id int, c1 text, c2 text, v int, primary key (id, c1,
> c2)) with compact storage and compression = {'sstable_compression': ''};
> {code}
> sstable2json before upgrading:
> {code}
> [
> {"key": "1",
> "cells": [["","0",1470761440040513],
>["a","asd",2470761440040513,"t",1470764842],
>["asd:","0",1470761451368658],
>["asd:asd","0",1470761449416613]]}
> ]
> {code}
> Query result with 2.1.9:
> {code}
> cqlsh> select * from ks1.test;
> id | c1 | c2 | v
> +-+--+---
> 1 | | null | 0
> 1 | asd | | 0
> 1 | asd | asd | 0
> (3 rows)
> {code}
> Query result with 3.7:
> {code}
> cqlsh> select * from ks1.test;
> id | 6331 | 6332 | v
> +--+--+---
> 1 | | null | 0
> (1 rows)
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (CASSANDRA-12423) Cells missing from compact storage table after upgrading from 2.1.9 to 3.7
[
https://issues.apache.org/jira/browse/CASSANDRA-12423?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15425798#comment-15425798
]
Stefania commented on CASSANDRA-12423:
--
The tests look better now, I've restarted the unit and dtests for trunk and the
dtests for 3.0 to shake off some unrelated failures that have been fixed in the
last couple of days.
I changed the initial approach and I've added a new legacy bound kind that
behaves like {{Composites.EOC.NONE}} does in 2.1: an inclusive END but it
compares before clustering values. This patch is ready for review, the issue
with the clustering column names is probably better dealt with in a separate
ticket if it needs fixing, ping [~iamaleksey].
> Cells missing from compact storage table after upgrading from 2.1.9 to 3.7
> --
>
> Key: CASSANDRA-12423
> URL: https://issues.apache.org/jira/browse/CASSANDRA-12423
> Project: Cassandra
> Issue Type: Bug
>Reporter: Tomasz Grabiec
>Assignee: Stefania
> Attachments: 12423.tar.gz
>
>
> Schema:
> {code}
> create table ks1.test ( id int, c1 text, c2 text, v int, primary key (id, c1,
> c2)) with compact storage and compression = {'sstable_compression': ''};
> {code}
> sstable2json before upgrading:
> {code}
> [
> {"key": "1",
> "cells": [["","0",1470761440040513],
>["a","asd",2470761440040513,"t",1470764842],
>["asd:","0",1470761451368658],
>["asd:asd","0",1470761449416613]]}
> ]
> {code}
> Query result with 2.1.9:
> {code}
> cqlsh> select * from ks1.test;
> id | c1 | c2 | v
> +-+--+---
> 1 | | null | 0
> 1 | asd | | 0
> 1 | asd | asd | 0
> (3 rows)
> {code}
> Query result with 3.7:
> {code}
> cqlsh> select * from ks1.test;
> id | 6331 | 6332 | v
> +--+--+---
> 1 | | null | 0
> (1 rows)
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (CASSANDRA-12423) Cells missing from compact storage table after upgrading from 2.1.9 to 3.7
[
https://issues.apache.org/jira/browse/CASSANDRA-12423?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15422003#comment-15422003
]
Stefania commented on CASSANDRA-12423:
--
Some existing thrift tests
[failed|http://cassci.datastax.com/job/stef1927-12423-3.0-dtest/1/testReport/],
resuming progress.
> Cells missing from compact storage table after upgrading from 2.1.9 to 3.7
> --
>
> Key: CASSANDRA-12423
> URL: https://issues.apache.org/jira/browse/CASSANDRA-12423
> Project: Cassandra
> Issue Type: Bug
>Reporter: Tomasz Grabiec
>Assignee: Stefania
> Attachments: 12423.tar.gz
>
>
> Schema:
> {code}
> create table ks1.test ( id int, c1 text, c2 text, v int, primary key (id, c1,
> c2)) with compact storage and compression = {'sstable_compression': ''};
> {code}
> sstable2json before upgrading:
> {code}
> [
> {"key": "1",
> "cells": [["","0",1470761440040513],
>["a","asd",2470761440040513,"t",1470764842],
>["asd:","0",1470761451368658],
>["asd:asd","0",1470761449416613]]}
> ]
> {code}
> Query result with 2.1.9:
> {code}
> cqlsh> select * from ks1.test;
> id | c1 | c2 | v
> +-+--+---
> 1 | | null | 0
> 1 | asd | | 0
> 1 | asd | asd | 0
> (3 rows)
> {code}
> Query result with 3.7:
> {code}
> cqlsh> select * from ks1.test;
> id | 6331 | 6332 | v
> +--+--+---
> 1 | | null | 0
> (1 rows)
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (CASSANDRA-12423) Cells missing from compact storage table after upgrading from 2.1.9 to 3.7
[
https://issues.apache.org/jira/browse/CASSANDRA-12423?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15420740#comment-15420740
]
Stefania commented on CASSANDRA-12423:
--
Thank you for the details. I've created 2 new tests
[here|https://github.com/stef1927/cassandra-dtest/commits/12423], one for
upgrading sstables from 2.1.9 to 3.0+ and one that just creates a range
tombstone with EOC=0. I've reproduced the problem in both cases.
The patch is in {{LegacyLayout}} as indicated above:
||3.0||3.9||trunk||
|[patch|https://github.com/stef1927/cassandra/commits/12423-3.0]|[patch|https://github.com/stef1927/cassandra/commits/12423-3.9]|[patch|https://github.com/stef1927/cassandra/commits/12423]|
|[testall|http://cassci.datastax.com/view/Dev/view/stef1927/job/stef1927-12423-3.0-testall/]|[testall|http://cassci.datastax.com/view/Dev/view/stef1927/job/stef1927-12423-3.9-testall/]|[testall|http://cassci.datastax.com/view/Dev/view/stef1927/job/stef1927-12423-testall/]|
|[dtest|http://cassci.datastax.com/view/Dev/view/stef1927/job/stef1927-12423-3.0-dtest/]|[dtest|http://cassci.datastax.com/view/Dev/view/stef1927/job/stef1927-12423-3.9-dtest/]|[dtest|http://cassci.datastax.com/view/Dev/view/stef1927/job/stef1927-12423-dtest/]|
Aside from this issue, there is then the problem that the clustering column
names "c1" and "c2" are converted into their hex bytes "6331" and "6332". I've
traced this to this code in {{LegacySchemaMigrator}}:
{code}
// Note: we save the column name as string, but we should not assume
that it is an UTF8 name, we
// we need to use the comparator fromString method
AbstractType comparator = isCQLTable
? UTF8Type.instance
:
CompactTables.columnDefinitionComparator(rawKind, isSuper, rawComparator,
rawSubComparator);
ColumnIdentifier name =
ColumnIdentifier.getInterned(comparator.fromString(row.getString("column_name")),
comparator);
{code}
The problem is that there is another table, {{Standard3}} created by
thrift_test.py, and in this case we create an interned column identifier with
comparator set to {{BytesType}}, which means the column name ends up being the
hex bytes corresponding to "c1": "6331", and similar for "c2". This by itself
seems wrong but maybe it is a known thrift limitation. Then, when we create the
identifier for our test table, even though the comparator here is correctly set
to UTF8, {{ColumnIdentifier.internedInstances}} ensures that we pick up the
same incorrect value because the UTF-8 encoding of "c1" or "c2" has the same
bytes. [~iamaleksey] is this something you are aware of or does it need fixing?
And if so, how can we ever rely on {{ColumnIdentifier.internedInstances}} if we
use non UTF8 comparators? My guess is that
{{CompactTables.columnDefinitionComparator}} is returning the wrong comparator
for {{Standard3}} but I'm not 100% sure.
I've attached the sstables generated with 2.1, [^12423.tar.gz], if you want to
reproduce it quickly, just start 3.0 with these sstables after setting the
partitioner and cluster name to {{ByteOrderedPartitioner}} and {{test}}
respectively.
> Cells missing from compact storage table after upgrading from 2.1.9 to 3.7
> --
>
> Key: CASSANDRA-12423
> URL: https://issues.apache.org/jira/browse/CASSANDRA-12423
> Project: Cassandra
> Issue Type: Bug
>Reporter: Tomasz Grabiec
>Assignee: Stefania
> Attachments: 12423.tar.gz
>
>
> Schema:
> {code}
> create table ks1.test ( id int, c1 text, c2 text, v int, primary key (id, c1,
> c2)) with compact storage and compression = {'sstable_compression': ''};
> {code}
> sstable2json before upgrading:
> {code}
> [
> {"key": "1",
> "cells": [["","0",1470761440040513],
>["a","asd",2470761440040513,"t",1470764842],
>["asd:","0",1470761451368658],
>["asd:asd","0",1470761449416613]]}
> ]
> {code}
> Query result with 2.1.9:
> {code}
> cqlsh> select * from ks1.test;
> id | c1 | c2 | v
> +-+--+---
> 1 | | null | 0
> 1 | asd | | 0
> 1 | asd | asd | 0
> (3 rows)
> {code}
> Query result with 3.7:
> {code}
> cqlsh> select * from ks1.test;
> id | 6331 | 6332 | v
> +--+--+---
> 1 | | null | 0
> (1 rows)
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (CASSANDRA-12423) Cells missing from compact storage table after upgrading from 2.1.9 to 3.7
[
https://issues.apache.org/jira/browse/CASSANDRA-12423?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15418636#comment-15418636
]
Tomasz Grabiec commented on CASSANDRA-12423:
The range delete was generated using Thrift API's SlicePredicate:
{code}
def composite(item1, item2=None, eoc='\x00'):
packed = _i16(len(item1)) + item1 + eoc
if item2 is not None:
packed += _i16(len(item2)) + item2
packed += eoc
return packed
def range_delete(self):
_set_keyspace('ks1')
mutations = [Mutation(deletion=Deletion(2470761440040513,
predicate=SlicePredicate(slice_range=SliceRange(start=composite('a',
eoc='\x00'), finish=composite('asd', eoc='\x00')]
keyed_mutations = {
_i32(1): {
'test': mutations
}
}
client.batch_mutate(keyed_mutations, ConsistencyLevel.ONE)
{code}
I executed above as part of thrift_tests.py from C* dtest.
> Cells missing from compact storage table after upgrading from 2.1.9 to 3.7
> --
>
> Key: CASSANDRA-12423
> URL: https://issues.apache.org/jira/browse/CASSANDRA-12423
> Project: Cassandra
> Issue Type: Bug
>Reporter: Tomasz Grabiec
>Assignee: Stefania
>
> Schema:
> {code}
> create table ks1.test ( id int, c1 text, c2 text, v int, primary key (id, c1,
> c2)) with compact storage and compression = {'sstable_compression': ''};
> {code}
> sstable2json before upgrading:
> {code}
> [
> {"key": "1",
> "cells": [["","0",1470761440040513],
>["a","asd",2470761440040513,"t",1470764842],
>["asd:","0",1470761451368658],
>["asd:asd","0",1470761449416613]]}
> ]
> {code}
> Query result with 2.1.9:
> {code}
> cqlsh> select * from ks1.test;
> id | c1 | c2 | v
> +-+--+---
> 1 | | null | 0
> 1 | asd | | 0
> 1 | asd | asd | 0
> (3 rows)
> {code}
> Query result with 3.7:
> {code}
> cqlsh> select * from ks1.test;
> id | 6331 | 6332 | v
> +--+--+---
> 1 | | null | 0
> (1 rows)
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (CASSANDRA-12423) Cells missing from compact storage table after upgrading from 2.1.9 to 3.7
[
https://issues.apache.org/jira/browse/CASSANDRA-12423?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15418548#comment-15418548
]
Stefania commented on CASSANDRA-12423:
--
I'm trying to create an sstable to reproduce the problem but I don't understand
how we can create a range tombstone with EOC=0.
The CQL delete statement uses {{AbstractComposite.slice()}}, which ensures that
the start has {{EOC=-1}} and the end has {{EOC=1}}. Alternatively it uses a
deleted cell. From what I could gather cassandra-cli also uses a deleted cell
for compact storage but I am really not familiar with Thrift so perhaps there
is a way to create a range tombstone with EOC=0 using cassandra-cli.
Do you still have the steps to create an sstable with this problem?
> Cells missing from compact storage table after upgrading from 2.1.9 to 3.7
> --
>
> Key: CASSANDRA-12423
> URL: https://issues.apache.org/jira/browse/CASSANDRA-12423
> Project: Cassandra
> Issue Type: Bug
>Reporter: Tomasz Grabiec
>Assignee: Stefania
>
> Schema:
> {code}
> create table ks1.test ( id int, c1 text, c2 text, v int, primary key (id, c1,
> c2)) with compact storage and compression = {'sstable_compression': ''};
> {code}
> sstable2json before upgrading:
> {code}
> [
> {"key": "1",
> "cells": [["","0",1470761440040513],
>["a","asd",2470761440040513,"t",1470764842],
>["asd:","0",1470761451368658],
>["asd:asd","0",1470761449416613]]}
> ]
> {code}
> Query result with 2.1.9:
> {code}
> cqlsh> select * from ks1.test;
> id | c1 | c2 | v
> +-+--+---
> 1 | | null | 0
> 1 | asd | | 0
> 1 | asd | asd | 0
> (3 rows)
> {code}
> Query result with 3.7:
> {code}
> cqlsh> select * from ks1.test;
> id | 6331 | 6332 | v
> +--+--+---
> 1 | | null | 0
> (1 rows)
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (CASSANDRA-12423) Cells missing from compact storage table after upgrading from 2.1.9 to 3.7
[
https://issues.apache.org/jira/browse/CASSANDRA-12423?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15414919#comment-15414919
]
Tomasz Grabiec commented on CASSANDRA-12423:
The problem could be that org.apache.cassandra.db.LegacyLayout#decodeBound
decodes upper bound of the range tombstone with eoc=0 the same as eoc=1
(INCL_END_BOUND) and both will cover all cells which are prefixed by the short
key. Whereas in 2.1.9, composite ordering orders eoc=0 bound before the cells
prefixed by it.
> Cells missing from compact storage table after upgrading from 2.1.9 to 3.7
> --
>
> Key: CASSANDRA-12423
> URL: https://issues.apache.org/jira/browse/CASSANDRA-12423
> Project: Cassandra
> Issue Type: Bug
>Reporter: Tomasz Grabiec
>
> Schema:
> {code}
> create table ks1.test ( id int, c1 text, c2 text, v int, primary key (id, c1,
> c2)) with compact storage and compression = {'sstable_compression': ''};
> {code}
> sstable2json before upgrading:
> {code}
> [
> {"key": "1",
> "cells": [["","0",1470761440040513],
>["a","asd",2470761440040513,"t",1470764842],
>["asd:","0",1470761451368658],
>["asd:asd","0",1470761449416613]]}
> ]
> {code}
> Query result with 2.1.9:
> {code}
> cqlsh> select * from ks1.test;
> id | c1 | c2 | v
> +-+--+---
> 1 | | null | 0
> 1 | asd | | 0
> 1 | asd | asd | 0
> (3 rows)
> {code}
> Query result with 3.7:
> {code}
> cqlsh> select * from ks1.test;
> id | 6331 | 6332 | v
> +--+--+---
> 1 | | null | 0
> (1 rows)
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
