[jira] [Commented] (CASSANDRA-14568) Static collection deletions are corrupted in 3.0 -> 2.{1,2} messages
[
https://issues.apache.org/jira/browse/CASSANDRA-14568?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16614518#comment-16614518
]
Sylvain Lebresne commented on CASSANDRA-14568:
--
+1 on this one.
> Static collection deletions are corrupted in 3.0 -> 2.{1,2} messages
>
>
> Key: CASSANDRA-14568
> URL: https://issues.apache.org/jira/browse/CASSANDRA-14568
> Project: Cassandra
> Issue Type: Bug
>Reporter: Benedict
>Assignee: Benedict
>Priority: Critical
> Fix For: 3.0.17, 3.11.3
>
>
> In 2.1 and 2.2, row and complex deletions were represented as range
> tombstones. LegacyLayout is our compatibility layer, that translates the
> relevant RT patterns in 2.1/2.2 to row/complex deletions in 3.0, and vice
> versa. Unfortunately, it does not handle the special case of static row
> deletions, they are treated as regular row deletions. Since static rows are
> themselves never directly deleted, the only issue is with collection
> deletions.
> Collection deletions in 2.1/2.2 were encoded as a range tombstone, consisting
> of a sequence of the clustering keys’ data for the affected row, followed by
> the bytes representing the name of the collection column. STATIC_CLUSTERING
> contains zero clusterings, so by treating the deletion as for a regular row,
> zero clusterings are written to precede the column name of the erased
> collection, so the column name is written at position zero.
> This can exhibit itself in at least two ways:
> # If the type of your first clustering key is a variable width type, new
> deletes will begin appearing covering the clustering key represented by the
> column name.
> ** If you have multiple clustering keys, you will receive a RT covering all
> those rows with a matching first clustering key.
> ** This RT will be valid as far as the system is concerned, and go
> undetected unless there are outside data quality checks in place.
> # Otherwise, an invalid size of data will be written to the clustering and
> sent over the network to the 2.1 node.
> ** The 2.1/2.2 node will handle this just fine, even though the record is
> junk. Since it is a deletion covering impossible data, there will be no
> user-API visible effect. But if received as a write from a 3.0 node, it will
> dutifully persist the junk record.
> ** The 3.0 node that originally sent this junk, may later coordinate a read
> of the partition, and will notice a digest mismatch, read-repair and
> serialize the junk to disk
> ** The sstable containing this record is now corrupt; the deserialization
> expects fixed-width data, but it encounters too many (or too few) bytes, and
> is now at an incorrect position to read its structural information
> ** (Alternatively when the 2.1 node is upgraded this will occur on eventual
> compaction)
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Commented] (CASSANDRA-14568) Static collection deletions are corrupted in 3.0 -> 2.{1,2} messages
[
https://issues.apache.org/jira/browse/CASSANDRA-14568?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16613724#comment-16613724
]
Benedict commented on CASSANDRA-14568:
--
[3.0|https://github.com/belliottsmith/cassandra/tree/14568]
[CI|https://circleci.com/workflow-run/70397395-3585-4b4c-904f-a55c070cf359]
Thanks both for your review. I have split out CASSANDRA-14749, and pushed an
updated patch simply missing this part. If either of you could give a quick
cursory +1, I'll commit them both.
> Static collection deletions are corrupted in 3.0 -> 2.{1,2} messages
>
>
> Key: CASSANDRA-14568
> URL: https://issues.apache.org/jira/browse/CASSANDRA-14568
> Project: Cassandra
> Issue Type: Bug
>Reporter: Benedict
>Assignee: Benedict
>Priority: Critical
> Fix For: 3.0.17, 3.11.3
>
>
> In 2.1 and 2.2, row and complex deletions were represented as range
> tombstones. LegacyLayout is our compatibility layer, that translates the
> relevant RT patterns in 2.1/2.2 to row/complex deletions in 3.0, and vice
> versa. Unfortunately, it does not handle the special case of static row
> deletions, they are treated as regular row deletions. Since static rows are
> themselves never directly deleted, the only issue is with collection
> deletions.
> Collection deletions in 2.1/2.2 were encoded as a range tombstone, consisting
> of a sequence of the clustering keys’ data for the affected row, followed by
> the bytes representing the name of the collection column. STATIC_CLUSTERING
> contains zero clusterings, so by treating the deletion as for a regular row,
> zero clusterings are written to precede the column name of the erased
> collection, so the column name is written at position zero.
> This can exhibit itself in at least two ways:
> # If the type of your first clustering key is a variable width type, new
> deletes will begin appearing covering the clustering key represented by the
> column name.
> ** If you have multiple clustering keys, you will receive a RT covering all
> those rows with a matching first clustering key.
> ** This RT will be valid as far as the system is concerned, and go
> undetected unless there are outside data quality checks in place.
> # Otherwise, an invalid size of data will be written to the clustering and
> sent over the network to the 2.1 node.
> ** The 2.1/2.2 node will handle this just fine, even though the record is
> junk. Since it is a deletion covering impossible data, there will be no
> user-API visible effect. But if received as a write from a 3.0 node, it will
> dutifully persist the junk record.
> ** The 3.0 node that originally sent this junk, may later coordinate a read
> of the partition, and will notice a digest mismatch, read-repair and
> serialize the junk to disk
> ** The sstable containing this record is now corrupt; the deserialization
> expects fixed-width data, but it encounters too many (or too few) bytes, and
> is now at an incorrect position to read its structural information
> ** (Alternatively when the 2.1 node is upgraded this will occur on eventual
> compaction)
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Commented] (CASSANDRA-14568) Static collection deletions are corrupted in 3.0 -> 2.{1,2} messages
[
https://issues.apache.org/jira/browse/CASSANDRA-14568?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16595319#comment-16595319
]
Aleksey Yeschenko commented on CASSANDRA-14568:
---
Looks good to me as well. Sorry for failing to spot this in the first place.
Old memories are fading.
> Static collection deletions are corrupted in 3.0 -> 2.{1,2} messages
>
>
> Key: CASSANDRA-14568
> URL: https://issues.apache.org/jira/browse/CASSANDRA-14568
> Project: Cassandra
> Issue Type: Bug
>Reporter: Benedict
>Assignee: Benedict
>Priority: Critical
> Fix For: 3.0.17, 3.11.3
>
>
> In 2.1 and 2.2, row and complex deletions were represented as range
> tombstones. LegacyLayout is our compatibility layer, that translates the
> relevant RT patterns in 2.1/2.2 to row/complex deletions in 3.0, and vice
> versa. Unfortunately, it does not handle the special case of static row
> deletions, they are treated as regular row deletions. Since static rows are
> themselves never directly deleted, the only issue is with collection
> deletions.
> Collection deletions in 2.1/2.2 were encoded as a range tombstone, consisting
> of a sequence of the clustering keys’ data for the affected row, followed by
> the bytes representing the name of the collection column. STATIC_CLUSTERING
> contains zero clusterings, so by treating the deletion as for a regular row,
> zero clusterings are written to precede the column name of the erased
> collection, so the column name is written at position zero.
> This can exhibit itself in at least two ways:
> # If the type of your first clustering key is a variable width type, new
> deletes will begin appearing covering the clustering key represented by the
> column name.
> ** If you have multiple clustering keys, you will receive a RT covering all
> those rows with a matching first clustering key.
> ** This RT will be valid as far as the system is concerned, and go
> undetected unless there are outside data quality checks in place.
> # Otherwise, an invalid size of data will be written to the clustering and
> sent over the network to the 2.1 node.
> ** The 2.1/2.2 node will handle this just fine, even though the record is
> junk. Since it is a deletion covering impossible data, there will be no
> user-API visible effect. But if received as a write from a 3.0 node, it will
> dutifully persist the junk record.
> ** The 3.0 node that originally sent this junk, may later coordinate a read
> of the partition, and will notice a digest mismatch, read-repair and
> serialize the junk to disk
> ** The sstable containing this record is now corrupt; the deserialization
> expects fixed-width data, but it encounters too many (or too few) bytes, and
> is now at an incorrect position to read its structural information
> ** (Alternatively when the 2.1 node is upgraded this will occur on eventual
> compaction)
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Commented] (CASSANDRA-14568) Static collection deletions are corrupted in 3.0 -> 2.{1,2} messages
[
https://issues.apache.org/jira/browse/CASSANDRA-14568?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16591837#comment-16591837
]
Sylvain Lebresne commented on CASSANDRA-14568:
--
bq. I may commit this ahead of then
Wfm. Maybe have [~iamaleksey] check our logic one more time but otherwise +1
from me.
> Static collection deletions are corrupted in 3.0 -> 2.{1,2} messages
>
>
> Key: CASSANDRA-14568
> URL: https://issues.apache.org/jira/browse/CASSANDRA-14568
> Project: Cassandra
> Issue Type: Bug
>Reporter: Benedict
>Assignee: Benedict
>Priority: Critical
> Fix For: 3.0.17, 3.11.3
>
>
> In 2.1 and 2.2, row and complex deletions were represented as range
> tombstones. LegacyLayout is our compatibility layer, that translates the
> relevant RT patterns in 2.1/2.2 to row/complex deletions in 3.0, and vice
> versa. Unfortunately, it does not handle the special case of static row
> deletions, they are treated as regular row deletions. Since static rows are
> themselves never directly deleted, the only issue is with collection
> deletions.
> Collection deletions in 2.1/2.2 were encoded as a range tombstone, consisting
> of a sequence of the clustering keys’ data for the affected row, followed by
> the bytes representing the name of the collection column. STATIC_CLUSTERING
> contains zero clusterings, so by treating the deletion as for a regular row,
> zero clusterings are written to precede the column name of the erased
> collection, so the column name is written at position zero.
> This can exhibit itself in at least two ways:
> # If the type of your first clustering key is a variable width type, new
> deletes will begin appearing covering the clustering key represented by the
> column name.
> ** If you have multiple clustering keys, you will receive a RT covering all
> those rows with a matching first clustering key.
> ** This RT will be valid as far as the system is concerned, and go
> undetected unless there are outside data quality checks in place.
> # Otherwise, an invalid size of data will be written to the clustering and
> sent over the network to the 2.1 node.
> ** The 2.1/2.2 node will handle this just fine, even though the record is
> junk. Since it is a deletion covering impossible data, there will be no
> user-API visible effect. But if received as a write from a 3.0 node, it will
> dutifully persist the junk record.
> ** The 3.0 node that originally sent this junk, may later coordinate a read
> of the partition, and will notice a digest mismatch, read-repair and
> serialize the junk to disk
> ** The sstable containing this record is now corrupt; the deserialization
> expects fixed-width data, but it encounters too many (or too few) bytes, and
> is now at an incorrect position to read its structural information
> ** (Alternatively when the 2.1 node is upgraded this will occur on eventual
> compaction)
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Commented] (CASSANDRA-14568) Static collection deletions are corrupted in 3.0 -> 2.{1,2} messages
[
https://issues.apache.org/jira/browse/CASSANDRA-14568?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16591674#comment-16591674
]
Benedict commented on CASSANDRA-14568:
--
Great, thanks for taking the time to confirm.
bq. upgrade dtests would be great to validate it.
Yep, absolutely. That will have to wait until things settle down in a couple
of weeks, after the freeze. I may commit this ahead of then, just so we have
the bug (at least much more probably) fixed for anyone who might need it more
urgently.
> Static collection deletions are corrupted in 3.0 -> 2.{1,2} messages
>
>
> Key: CASSANDRA-14568
> URL: https://issues.apache.org/jira/browse/CASSANDRA-14568
> Project: Cassandra
> Issue Type: Bug
>Reporter: Benedict
>Assignee: Benedict
>Priority: Critical
> Fix For: 3.0.17, 3.11.3
>
>
> In 2.1 and 2.2, row and complex deletions were represented as range
> tombstones. LegacyLayout is our compatibility layer, that translates the
> relevant RT patterns in 2.1/2.2 to row/complex deletions in 3.0, and vice
> versa. Unfortunately, it does not handle the special case of static row
> deletions, they are treated as regular row deletions. Since static rows are
> themselves never directly deleted, the only issue is with collection
> deletions.
> Collection deletions in 2.1/2.2 were encoded as a range tombstone, consisting
> of a sequence of the clustering keys’ data for the affected row, followed by
> the bytes representing the name of the collection column. STATIC_CLUSTERING
> contains zero clusterings, so by treating the deletion as for a regular row,
> zero clusterings are written to precede the column name of the erased
> collection, so the column name is written at position zero.
> This can exhibit itself in at least two ways:
> # If the type of your first clustering key is a variable width type, new
> deletes will begin appearing covering the clustering key represented by the
> column name.
> ** If you have multiple clustering keys, you will receive a RT covering all
> those rows with a matching first clustering key.
> ** This RT will be valid as far as the system is concerned, and go
> undetected unless there are outside data quality checks in place.
> # Otherwise, an invalid size of data will be written to the clustering and
> sent over the network to the 2.1 node.
> ** The 2.1/2.2 node will handle this just fine, even though the record is
> junk. Since it is a deletion covering impossible data, there will be no
> user-API visible effect. But if received as a write from a 3.0 node, it will
> dutifully persist the junk record.
> ** The 3.0 node that originally sent this junk, may later coordinate a read
> of the partition, and will notice a digest mismatch, read-repair and
> serialize the junk to disk
> ** The sstable containing this record is now corrupt; the deserialization
> expects fixed-width data, but it encounters too many (or too few) bytes, and
> is now at an incorrect position to read its structural information
> ** (Alternatively when the 2.1 node is upgraded this will occur on eventual
> compaction)
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Commented] (CASSANDRA-14568) Static collection deletions are corrupted in 3.0 -> 2.{1,2} messages
[
https://issues.apache.org/jira/browse/CASSANDRA-14568?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16591668#comment-16591668
]
Sylvain Lebresne commented on CASSANDRA-14568:
--
Sorry for the slow turn over. Last version lgtm, though as mention, upgrade
dtests would be great to validate it.
Re: dropped static collection columns, the bug looks legit as well so kudos for
noticing that. And I personally don't mind committing it here directly.
> Static collection deletions are corrupted in 3.0 -> 2.{1,2} messages
>
>
> Key: CASSANDRA-14568
> URL: https://issues.apache.org/jira/browse/CASSANDRA-14568
> Project: Cassandra
> Issue Type: Bug
>Reporter: Benedict
>Assignee: Benedict
>Priority: Critical
> Fix For: 3.0.17, 3.11.3
>
>
> In 2.1 and 2.2, row and complex deletions were represented as range
> tombstones. LegacyLayout is our compatibility layer, that translates the
> relevant RT patterns in 2.1/2.2 to row/complex deletions in 3.0, and vice
> versa. Unfortunately, it does not handle the special case of static row
> deletions, they are treated as regular row deletions. Since static rows are
> themselves never directly deleted, the only issue is with collection
> deletions.
> Collection deletions in 2.1/2.2 were encoded as a range tombstone, consisting
> of a sequence of the clustering keys’ data for the affected row, followed by
> the bytes representing the name of the collection column. STATIC_CLUSTERING
> contains zero clusterings, so by treating the deletion as for a regular row,
> zero clusterings are written to precede the column name of the erased
> collection, so the column name is written at position zero.
> This can exhibit itself in at least two ways:
> # If the type of your first clustering key is a variable width type, new
> deletes will begin appearing covering the clustering key represented by the
> column name.
> ** If you have multiple clustering keys, you will receive a RT covering all
> those rows with a matching first clustering key.
> ** This RT will be valid as far as the system is concerned, and go
> undetected unless there are outside data quality checks in place.
> # Otherwise, an invalid size of data will be written to the clustering and
> sent over the network to the 2.1 node.
> ** The 2.1/2.2 node will handle this just fine, even though the record is
> junk. Since it is a deletion covering impossible data, there will be no
> user-API visible effect. But if received as a write from a 3.0 node, it will
> dutifully persist the junk record.
> ** The 3.0 node that originally sent this junk, may later coordinate a read
> of the partition, and will notice a digest mismatch, read-repair and
> serialize the junk to disk
> ** The sstable containing this record is now corrupt; the deserialization
> expects fixed-width data, but it encounters too many (or too few) bytes, and
> is now at an incorrect position to read its structural information
> ** (Alternatively when the 2.1 node is upgraded this will occur on eventual
> compaction)
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Commented] (CASSANDRA-14568) Static collection deletions are corrupted in 3.0 -> 2.{1,2} messages
[
https://issues.apache.org/jira/browse/CASSANDRA-14568?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16582286#comment-16582286
]
Benedict commented on CASSANDRA-14568:
--
OK. So, [here|https://github.com/belliottsmith/cassandra/commits/14568-2] is
may second attempt at fixing this.
In the process of adding improved assertion logic, I realised we might have
another bug around dropped static collection column, that could have resulted
in decoding a static collection deletion as a whole-static-row deletion (with
unknown semantics, since I vaguely recall that our correctness in some places
depends on there being no such deletions).
In essence: if on looking up the collectionNameBytes, we found no
collectionName (due, for instance, to it having been dropped), we would be left
with a only a complete static row bound to construct.
We clearly need to introduce some upgrade dtests to cover these cases as well
> Static collection deletions are corrupted in 3.0 -> 2.{1,2} messages
>
>
> Key: CASSANDRA-14568
> URL: https://issues.apache.org/jira/browse/CASSANDRA-14568
> Project: Cassandra
> Issue Type: Bug
>Reporter: Benedict
>Assignee: Benedict
>Priority: Critical
> Fix For: 3.0.17, 3.11.3
>
>
> In 2.1 and 2.2, row and complex deletions were represented as range
> tombstones. LegacyLayout is our compatibility layer, that translates the
> relevant RT patterns in 2.1/2.2 to row/complex deletions in 3.0, and vice
> versa. Unfortunately, it does not handle the special case of static row
> deletions, they are treated as regular row deletions. Since static rows are
> themselves never directly deleted, the only issue is with collection
> deletions.
> Collection deletions in 2.1/2.2 were encoded as a range tombstone, consisting
> of a sequence of the clustering keys’ data for the affected row, followed by
> the bytes representing the name of the collection column. STATIC_CLUSTERING
> contains zero clusterings, so by treating the deletion as for a regular row,
> zero clusterings are written to precede the column name of the erased
> collection, so the column name is written at position zero.
> This can exhibit itself in at least two ways:
> # If the type of your first clustering key is a variable width type, new
> deletes will begin appearing covering the clustering key represented by the
> column name.
> ** If you have multiple clustering keys, you will receive a RT covering all
> those rows with a matching first clustering key.
> ** This RT will be valid as far as the system is concerned, and go
> undetected unless there are outside data quality checks in place.
> # Otherwise, an invalid size of data will be written to the clustering and
> sent over the network to the 2.1 node.
> ** The 2.1/2.2 node will handle this just fine, even though the record is
> junk. Since it is a deletion covering impossible data, there will be no
> user-API visible effect. But if received as a write from a 3.0 node, it will
> dutifully persist the junk record.
> ** The 3.0 node that originally sent this junk, may later coordinate a read
> of the partition, and will notice a digest mismatch, read-repair and
> serialize the junk to disk
> ** The sstable containing this record is now corrupt; the deserialization
> expects fixed-width data, but it encounters too many (or too few) bytes, and
> is now at an incorrect position to read its structural information
> ** (Alternatively when the 2.1 node is upgraded this will occur on eventual
> compaction)
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Commented] (CASSANDRA-14568) Static collection deletions are corrupted in 3.0 -> 2.{1,2} messages
[
https://issues.apache.org/jira/browse/CASSANDRA-14568?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16582245#comment-16582245
]
Benedict commented on CASSANDRA-14568:
--
bq. My concern was that LRT.start.bound is directly referenced in
UnfilteredDeserializer when converting to a RTMarker.
Hmm. Thinking on it some more, I guess this is not a problem due to the fact
that we never (in any extant version) actually issue any deletions that (in
3.0) would be represented as RTs spanning static rows, so the problematic cases
*should* all be converted to collection tombstones only. I will add some
comments to LegacyLayout elaborating the inconsistencies of modern/legacy
static clusterings as part of the patch.
So, I'm now comfortable with fixing either location, I think. Though I need to
code dive a bit more to be absolutely certain.
> Static collection deletions are corrupted in 3.0 -> 2.{1,2} messages
>
>
> Key: CASSANDRA-14568
> URL: https://issues.apache.org/jira/browse/CASSANDRA-14568
> Project: Cassandra
> Issue Type: Bug
>Reporter: Benedict
>Assignee: Benedict
>Priority: Critical
> Fix For: 3.0.17, 3.11.3
>
>
> In 2.1 and 2.2, row and complex deletions were represented as range
> tombstones. LegacyLayout is our compatibility layer, that translates the
> relevant RT patterns in 2.1/2.2 to row/complex deletions in 3.0, and vice
> versa. Unfortunately, it does not handle the special case of static row
> deletions, they are treated as regular row deletions. Since static rows are
> themselves never directly deleted, the only issue is with collection
> deletions.
> Collection deletions in 2.1/2.2 were encoded as a range tombstone, consisting
> of a sequence of the clustering keys’ data for the affected row, followed by
> the bytes representing the name of the collection column. STATIC_CLUSTERING
> contains zero clusterings, so by treating the deletion as for a regular row,
> zero clusterings are written to precede the column name of the erased
> collection, so the column name is written at position zero.
> This can exhibit itself in at least two ways:
> # If the type of your first clustering key is a variable width type, new
> deletes will begin appearing covering the clustering key represented by the
> column name.
> ** If you have multiple clustering keys, you will receive a RT covering all
> those rows with a matching first clustering key.
> ** This RT will be valid as far as the system is concerned, and go
> undetected unless there are outside data quality checks in place.
> # Otherwise, an invalid size of data will be written to the clustering and
> sent over the network to the 2.1 node.
> ** The 2.1/2.2 node will handle this just fine, even though the record is
> junk. Since it is a deletion covering impossible data, there will be no
> user-API visible effect. But if received as a write from a 3.0 node, it will
> dutifully persist the junk record.
> ** The 3.0 node that originally sent this junk, may later coordinate a read
> of the partition, and will notice a digest mismatch, read-repair and
> serialize the junk to disk
> ** The sstable containing this record is now corrupt; the deserialization
> expects fixed-width data, but it encounters too many (or too few) bytes, and
> is now at an incorrect position to read its structural information
> ** (Alternatively when the 2.1 node is upgraded this will occur on eventual
> compaction)
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Commented] (CASSANDRA-14568) Static collection deletions are corrupted in 3.0 -> 2.{1,2} messages
[
https://issues.apache.org/jira/browse/CASSANDRA-14568?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16582147#comment-16582147
]
Benedict commented on CASSANDRA-14568:
--
Wouldn't the "more correct" place to change it be [in the bound
construction|https://github.com/apache/cassandra/blob/cassandra-3.0/src/java/org/apache/cassandra/db/LegacyLayout.java#L803]?
Presumably the LegacyBound.bound property should be correctly legacy-fied so
that any other LegacyLayout code correctly interprets / sorts it? This is what
I was planning on with [my wip
branch|https://github.com/belliottsmith/cassandra/commit/5b6742a7a6104bbd88d784db4d3bf7cd990cf057#diff-1a4af2aebd51e301cf0e73126722a8a4R805].
My concern was that LRT.start.bound is [directly referenced in
UnfilteredDeserializer when converting to a
RTMarker|https://github.com/belliottsmith/cassandra/blob/14568-2/src/java/org/apache/cassandra/db/UnfilteredDeserializer.java#L728].
I realise in the light of morning that this would not be a problem during
_serialization_, but maybe this is another bug in deserialization?
> Static collection deletions are corrupted in 3.0 -> 2.{1,2} messages
>
>
> Key: CASSANDRA-14568
> URL: https://issues.apache.org/jira/browse/CASSANDRA-14568
> Project: Cassandra
> Issue Type: Bug
>Reporter: Benedict
>Assignee: Benedict
>Priority: Critical
> Fix For: 3.0.17, 3.11.3
>
>
> In 2.1 and 2.2, row and complex deletions were represented as range
> tombstones. LegacyLayout is our compatibility layer, that translates the
> relevant RT patterns in 2.1/2.2 to row/complex deletions in 3.0, and vice
> versa. Unfortunately, it does not handle the special case of static row
> deletions, they are treated as regular row deletions. Since static rows are
> themselves never directly deleted, the only issue is with collection
> deletions.
> Collection deletions in 2.1/2.2 were encoded as a range tombstone, consisting
> of a sequence of the clustering keys’ data for the affected row, followed by
> the bytes representing the name of the collection column. STATIC_CLUSTERING
> contains zero clusterings, so by treating the deletion as for a regular row,
> zero clusterings are written to precede the column name of the erased
> collection, so the column name is written at position zero.
> This can exhibit itself in at least two ways:
> # If the type of your first clustering key is a variable width type, new
> deletes will begin appearing covering the clustering key represented by the
> column name.
> ** If you have multiple clustering keys, you will receive a RT covering all
> those rows with a matching first clustering key.
> ** This RT will be valid as far as the system is concerned, and go
> undetected unless there are outside data quality checks in place.
> # Otherwise, an invalid size of data will be written to the clustering and
> sent over the network to the 2.1 node.
> ** The 2.1/2.2 node will handle this just fine, even though the record is
> junk. Since it is a deletion covering impossible data, there will be no
> user-API visible effect. But if received as a write from a 3.0 node, it will
> dutifully persist the junk record.
> ** The 3.0 node that originally sent this junk, may later coordinate a read
> of the partition, and will notice a digest mismatch, read-repair and
> serialize the junk to disk
> ** The sstable containing this record is now corrupt; the deserialization
> expects fixed-width data, but it encounters too many (or too few) bytes, and
> is now at an incorrect position to read its structural information
> ** (Alternatively when the 2.1 node is upgraded this will occur on eventual
> compaction)
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Commented] (CASSANDRA-14568) Static collection deletions are corrupted in 3.0 -> 2.{1,2} messages
[
https://issues.apache.org/jira/browse/CASSANDRA-14568?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16582108#comment-16582108
]
Sylvain Lebresne commented on CASSANDRA-14568:
--
It's absolutely possible I'm missing some problem, as god knows those backward
compatibility issues often comes with nasty surprises, but I don't think the
problem is too hard to solve.
That is, {{LegacyBound}} is only here for backward compatibility and never used
by 3.0 before being converting first, so I'm sure to understand what you mean
by "(setting it) to something that breaks the 3.0 definition of a static
clustering".
I believe we should simply make sure that when encoding to 2.x a
{{STATIC_CLUSTERING}}, we append "clustering size" empty values before adding
the rest (the column name), and when decoding, we assume those empty values are
here (but ignore them). This is what we do for "cell names"
([here|https://github.com/apache/cassandra/blob/cassandra-3.0/src/java/org/apache/cassandra/db/LegacyLayout.java#L287-L291]
and
[here|https://github.com/apache/cassandra/blob/cassandra-3.0/src/java/org/apache/cassandra/db/LegacyLayout.java#L329-L330])
and to the best of my knowledge is working properly.
In other words, I "think" we need to do the 2 following changes:
# revert the change made by this patch in
[{{LegacyLayout.decodeBound}}|https://github.com/apache/cassandra/blob/cassandra-3.0/src/java/org/apache/cassandra/db/LegacyLayout.java#L210-L215].
# make sure that in
[{{LegacyLayout.serializeCompound}}|https://github.com/apache/cassandra/blob/cassandra-3.0/src/java/org/apache/cassandra/db/LegacyLayout.java#L2394-L2400]
(and {{serializeSizeCompound}}), the static case is specialized to
automatically add the empty values, which might possibly be better handler
directly within {{CompositeType.Builder}}.
> Static collection deletions are corrupted in 3.0 -> 2.{1,2} messages
>
>
> Key: CASSANDRA-14568
> URL: https://issues.apache.org/jira/browse/CASSANDRA-14568
> Project: Cassandra
> Issue Type: Bug
>Reporter: Benedict
>Assignee: Benedict
>Priority: Critical
> Fix For: 3.0.17, 3.11.3
>
>
> In 2.1 and 2.2, row and complex deletions were represented as range
> tombstones. LegacyLayout is our compatibility layer, that translates the
> relevant RT patterns in 2.1/2.2 to row/complex deletions in 3.0, and vice
> versa. Unfortunately, it does not handle the special case of static row
> deletions, they are treated as regular row deletions. Since static rows are
> themselves never directly deleted, the only issue is with collection
> deletions.
> Collection deletions in 2.1/2.2 were encoded as a range tombstone, consisting
> of a sequence of the clustering keys’ data for the affected row, followed by
> the bytes representing the name of the collection column. STATIC_CLUSTERING
> contains zero clusterings, so by treating the deletion as for a regular row,
> zero clusterings are written to precede the column name of the erased
> collection, so the column name is written at position zero.
> This can exhibit itself in at least two ways:
> # If the type of your first clustering key is a variable width type, new
> deletes will begin appearing covering the clustering key represented by the
> column name.
> ** If you have multiple clustering keys, you will receive a RT covering all
> those rows with a matching first clustering key.
> ** This RT will be valid as far as the system is concerned, and go
> undetected unless there are outside data quality checks in place.
> # Otherwise, an invalid size of data will be written to the clustering and
> sent over the network to the 2.1 node.
> ** The 2.1/2.2 node will handle this just fine, even though the record is
> junk. Since it is a deletion covering impossible data, there will be no
> user-API visible effect. But if received as a write from a 3.0 node, it will
> dutifully persist the junk record.
> ** The 3.0 node that originally sent this junk, may later coordinate a read
> of the partition, and will notice a digest mismatch, read-repair and
> serialize the junk to disk
> ** The sstable containing this record is now corrupt; the deserialization
> expects fixed-width data, but it encounters too many (or too few) bytes, and
> is now at an incorrect position to read its structural information
> ** (Alternatively when the 2.1 node is upgraded this will occur on eventual
> compaction)
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Commented] (CASSANDRA-14568) Static collection deletions are corrupted in 3.0 -> 2.{1,2} messages
[
https://issues.apache.org/jira/browse/CASSANDRA-14568?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16581429#comment-16581429
]
Benedict commented on CASSANDRA-14568:
--
Thanks for the input, when fixing this I had indeed assumed the issue was only
with deserialization on 3.0, not serialization also.
I can now see there is an existing method for creating a staticBound, although
it's not presently clear to me if it was ever plausibly invoked, so it will
take a while to corroborate that we don't break any other assumptions by
setting the LegacyLayout.bound to something that breaks the 3.0 definition of a
static clustering. AFAICT, we do directly reference it in places where the
distinction matters, so we may have to introduce a legacyBound and modernBound
for which a distinction is only made in the static case.
> Static collection deletions are corrupted in 3.0 -> 2.{1,2} messages
>
>
> Key: CASSANDRA-14568
> URL: https://issues.apache.org/jira/browse/CASSANDRA-14568
> Project: Cassandra
> Issue Type: Bug
>Reporter: Benedict
>Assignee: Benedict
>Priority: Critical
> Fix For: 3.0.17, 3.11.3
>
>
> In 2.1 and 2.2, row and complex deletions were represented as range
> tombstones. LegacyLayout is our compatibility layer, that translates the
> relevant RT patterns in 2.1/2.2 to row/complex deletions in 3.0, and vice
> versa. Unfortunately, it does not handle the special case of static row
> deletions, they are treated as regular row deletions. Since static rows are
> themselves never directly deleted, the only issue is with collection
> deletions.
> Collection deletions in 2.1/2.2 were encoded as a range tombstone, consisting
> of a sequence of the clustering keys’ data for the affected row, followed by
> the bytes representing the name of the collection column. STATIC_CLUSTERING
> contains zero clusterings, so by treating the deletion as for a regular row,
> zero clusterings are written to precede the column name of the erased
> collection, so the column name is written at position zero.
> This can exhibit itself in at least two ways:
> # If the type of your first clustering key is a variable width type, new
> deletes will begin appearing covering the clustering key represented by the
> column name.
> ** If you have multiple clustering keys, you will receive a RT covering all
> those rows with a matching first clustering key.
> ** This RT will be valid as far as the system is concerned, and go
> undetected unless there are outside data quality checks in place.
> # Otherwise, an invalid size of data will be written to the clustering and
> sent over the network to the 2.1 node.
> ** The 2.1/2.2 node will handle this just fine, even though the record is
> junk. Since it is a deletion covering impossible data, there will be no
> user-API visible effect. But if received as a write from a 3.0 node, it will
> dutifully persist the junk record.
> ** The 3.0 node that originally sent this junk, may later coordinate a read
> of the partition, and will notice a digest mismatch, read-repair and
> serialize the junk to disk
> ** The sstable containing this record is now corrupt; the deserialization
> expects fixed-width data, but it encounters too many (or too few) bytes, and
> is now at an incorrect position to read its structural information
> ** (Alternatively when the 2.1 node is upgraded this will occur on eventual
> compaction)
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Commented] (CASSANDRA-14568) Static collection deletions are corrupted in 3.0 -> 2.{1,2} messages
[
https://issues.apache.org/jira/browse/CASSANDRA-14568?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16545537#comment-16545537
]
Benedict commented on CASSANDRA-14568:
--
Committed as d52c7b8c59, 31d5d870f9 and d3a994b105
tests were run against
[circleci|https://circleci.com/workflow-run/c818c58a-83e3-4731-89d7-ab0b04d26d62]
(unfortunately right now this link is down); this showed some failing auth
dtests, but these code paths were unaffected by this patch and I confirmed they
pass locally.
> Static collection deletions are corrupted in 3.0 -> 2.{1,2} messages
>
>
> Key: CASSANDRA-14568
> URL: https://issues.apache.org/jira/browse/CASSANDRA-14568
> Project: Cassandra
> Issue Type: Bug
>Reporter: Benedict
>Assignee: Benedict
>Priority: Critical
> Fix For: 3.0.x, 3.11.x
>
>
> In 2.1 and 2.2, row and complex deletions were represented as range
> tombstones. LegacyLayout is our compatibility layer, that translates the
> relevant RT patterns in 2.1/2.2 to row/complex deletions in 3.0, and vice
> versa. Unfortunately, it does not handle the special case of static row
> deletions, they are treated as regular row deletions. Since static rows are
> themselves never directly deleted, the only issue is with collection
> deletions.
> Collection deletions in 2.1/2.2 were encoded as a range tombstone, consisting
> of a sequence of the clustering keys’ data for the affected row, followed by
> the bytes representing the name of the collection column. STATIC_CLUSTERING
> contains zero clusterings, so by treating the deletion as for a regular row,
> zero clusterings are written to precede the column name of the erased
> collection, so the column name is written at position zero.
> This can exhibit itself in at least two ways:
> # If the type of your first clustering key is a variable width type, new
> deletes will begin appearing covering the clustering key represented by the
> column name.
> ** If you have multiple clustering keys, you will receive a RT covering all
> those rows with a matching first clustering key.
> ** This RT will be valid as far as the system is concerned, and go
> undetected unless there are outside data quality checks in place.
> # Otherwise, an invalid size of data will be written to the clustering and
> sent over the network to the 2.1 node.
> ** The 2.1/2.2 node will handle this just fine, even though the record is
> junk. Since it is a deletion covering impossible data, there will be no
> user-API visible effect. But if received as a write from a 3.0 node, it will
> dutifully persist the junk record.
> ** The 3.0 node that originally sent this junk, may later coordinate a read
> of the partition, and will notice a digest mismatch, read-repair and
> serialize the junk to disk
> ** The sstable containing this record is now corrupt; the deserialization
> expects fixed-width data, but it encounters too many (or too few) bytes, and
> is now at an incorrect position to read its structural information
> ** (Alternatively when the 2.1 node is upgraded this will occur on eventual
> compaction)
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Commented] (CASSANDRA-14568) Static collection deletions are corrupted in 3.0 -> 2.{1,2} messages
[
https://issues.apache.org/jira/browse/CASSANDRA-14568?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16545241#comment-16545241
]
Aleksey Yeschenko commented on CASSANDRA-14568:
---
+1, ship it.
> Static collection deletions are corrupted in 3.0 -> 2.{1,2} messages
>
>
> Key: CASSANDRA-14568
> URL: https://issues.apache.org/jira/browse/CASSANDRA-14568
> Project: Cassandra
> Issue Type: Bug
>Reporter: Benedict
>Assignee: Benedict
>Priority: Critical
> Fix For: 3.0.18
>
>
> In 2.1 and 2.2, row and complex deletions were represented as range
> tombstones. LegacyLayout is our compatibility layer, that translates the
> relevant RT patterns in 2.1/2.2 to row/complex deletions in 3.0, and vice
> versa. Unfortunately, it does not handle the special case of static row
> deletions, they are treated as regular row deletions. Since static rows are
> themselves never directly deleted, the only issue is with collection
> deletions.
> Collection deletions in 2.1/2.2 were encoded as a range tombstone, consisting
> of a sequence of the clustering keys’ data for the affected row, followed by
> the bytes representing the name of the collection column. STATIC_CLUSTERING
> contains zero clusterings, so by treating the deletion as for a regular row,
> zero clusterings are written to precede the column name of the erased
> collection, so the column name is written at position zero.
> This can exhibit itself in at least two ways:
> # If the type of your first clustering key is a variable width type, new
> deletes will begin appearing covering the clustering key represented by the
> column name.
> ** If you have multiple clustering keys, you will receive a RT covering all
> those rows with a matching first clustering key.
> ** This RT will be valid as far as the system is concerned, and go
> undetected unless there are outside data quality checks in place.
> # Otherwise, an invalid size of data will be written to the clustering and
> sent over the network to the 2.1 node.
> ** The 2.1/2.2 node will handle this just fine, even though the record is
> junk. Since it is a deletion covering impossible data, there will be no
> user-API visible effect. But if received as a write from a 3.0 node, it will
> dutifully persist the junk record.
> ** The 3.0 node that originally sent this junk, may later coordinate a read
> of the partition, and will notice a digest mismatch, read-repair and
> serialize the junk to disk
> ** The sstable containing this record is now corrupt; the deserialization
> expects fixed-width data, but it encounters too many (or too few) bytes, and
> is now at an incorrect position to read its structural information
> ** (Alternatively when the 2.1 node is upgraded this will occur on eventual
> compaction)
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Commented] (CASSANDRA-14568) Static collection deletions are corrupted in 3.0 -> 2.{1,2} messages
[
https://issues.apache.org/jira/browse/CASSANDRA-14568?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16543875#comment-16543875
]
Benedict commented on CASSANDRA-14568:
--
Patch available
[here|https://github.com/belliottsmith/cassandra/tree/CASSANDRA-14568]
> Static collection deletions are corrupted in 3.0 -> 2.{1,2} messages
>
>
> Key: CASSANDRA-14568
> URL: https://issues.apache.org/jira/browse/CASSANDRA-14568
> Project: Cassandra
> Issue Type: Bug
>Reporter: Benedict
>Assignee: Benedict
>Priority: Critical
> Fix For: 3.0.18
>
>
> In 2.1 and 2.2, row and complex deletions were represented as range
> tombstones. LegacyLayout is our compatibility layer, that translates the
> relevant RT patterns in 2.1/2.2 to row/complex deletions in 3.0, and vice
> versa. Unfortunately, it does not handle the special case of static row
> deletions, they are treated as regular row deletions. Since static rows are
> themselves never directly deleted, the only issue is with collection
> deletions.
> Collection deletions in 2.1/2.2 were encoded as a range tombstone, consisting
> of a sequence of the clustering keys’ data for the affected row, followed by
> the bytes representing the name of the collection column. STATIC_CLUSTERING
> contains zero clusterings, so by treating the deletion as for a regular row,
> zero clusterings are written to precede the column name of the erased
> collection, so the column name is written at position zero.
> This can exhibit itself in at least two ways:
> # If the type of your first clustering key is a variable width type, new
> deletes will begin appearing covering the clustering key represented by the
> column name.
> ** If you have multiple clustering keys, you will receive a RT covering all
> those rows with a matching first clustering key.
> ** This RT will be valid as far as the system is concerned, and go
> undetected unless there are outside data quality checks in place.
> # Otherwise, an invalid size of data will be written to the clustering and
> sent over the network to the 2.1 node.
> ** The 2.1/2.2 node will handle this just fine, even though the record is
> junk. Since it is a deletion covering impossible data, there will be no
> user-API visible effect. But if received as a write from a 3.0 node, it will
> dutifully persist the junk record.
> ** The 3.0 node that originally sent this junk, may later coordinate a read
> of the partition, and will notice a digest mismatch, read-repair and
> serialize the junk to disk
> ** The sstable containing this record is now corrupt; the deserialization
> expects fixed-width data, but it encounters too many (or too few) bytes, and
> is now at an incorrect position to read its structural information
> ** (Alternatively when the 2.1 node is upgraded this will occur on eventual
> compaction)
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org
