[jira] [Commented] (CASSANDRA-15789) Rows can get duplicated in mixed major-version clusters and after full upgrade
[
https://issues.apache.org/jira/browse/CASSANDRA-15789?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17226467#comment-17226467
]
Leon Zaruvinsky commented on CASSANDRA-15789:
-
Hey all - just wanted to call out that an errant import made it into the
Cassandra 3.11 patch of this PR:
https://github.com/apache/cassandra/blob/4d42c189fa82b32fd93ae42a164b91e4db62992e/src/java/org/apache/cassandra/service/StorageProxyMBean.java#L24
Not a big deal, but noticed this because we publish the MBeans/jmx api as their
own custom-built package and hit a compilation error because the
DatabaseDescriptor wasn't included.
> Rows can get duplicated in mixed major-version clusters and after full upgrade
> --
>
> Key: CASSANDRA-15789
> URL: https://issues.apache.org/jira/browse/CASSANDRA-15789
> Project: Cassandra
> Issue Type: Bug
> Components: Consistency/Coordination, Local/Memtable, Local/SSTable
>Reporter: Aleksey Yeschenko
>Assignee: Marcus Eriksson
>Priority: Normal
> Fix For: 3.0.21, 3.11.7, 4.0, 4.0-beta1
>
>
> In a mixed 2.X/3.X major version cluster a sequence of row deletes,
> collection overwrites, paging, and read repair can cause 3.X nodes to split
> individual rows into several rows with identical clustering. This happens due
> to 2.X paging and RT semantics, and a 3.X {{LegacyLayout}} deficiency.
> To reproduce, set up a 2-node mixed major version cluster with the following
> table:
> {code}
> CREATE TABLE distributed_test_keyspace.tlb (
> pk int,
> ck int,
> v map,
> PRIMARY KEY (pk, ck)
> );
> {code}
> 1. Using either node as the coordinator, delete the row with ck=2 using
> timestamp 1
> {code}
> DELETE FROM tbl USING TIMESTAMP 1 WHERE pk = 1 AND ck = 2;
> {code}
> 2. Using either node as the coordinator, insert the following 3 rows:
> {code}
> INSERT INTO tbl (pk, ck, v) VALUES (1, 1, {'e':'f'}) USING TIMESTAMP 3;
> INSERT INTO tbl (pk, ck, v) VALUES (1, 2, {'g':'h'}) USING TIMESTAMP 3;
> INSERT INTO tbl (pk, ck, v) VALUES (1, 3, {'i':'j'}) USING TIMESTAMP 3;
> {code}
> 3. Flush the table on both nodes
> 4. Using the 2.2 node as the coordinator, force read repar by querying the
> table with page size = 2:
>
> {code}
> SELECT * FROM tbl;
> {code}
> 5. Overwrite the row with ck=2 using timestamp 5:
> {code}
> INSERT INTO tbl (pk, ck, v) VALUES (1, 2, {'g':'h'}) USING TIMESTAMP 5;}}
> {code}
> 6. Query the 3.0 node and observe the split row:
> {code}
> cqlsh> select * from distributed_test_keyspace.tlb ;
> pk | ck | v
> ++
> 1 | 1 | {'e': 'f'}
> 1 | 2 | {'g': 'h'}
> 1 | 2 | {'k': 'l'}
> 1 | 3 | {'i': 'j'}
> {code}
> This happens because the read to query the second page ends up generating the
> following mutation for the 3.0 node:
> {code}
> ColumnFamily(tbl -{deletedAt=-9223372036854775808, localDeletion=2147483647,
> ranges=[2:v:_-2:v:!, deletedAt=2, localDeletion=1588588821]
> [2:v:!-2:!, deletedAt=1, localDeletion=1588588821]
> [3:v:_-3:v:!, deletedAt=2, localDeletion=1588588821]}-
> [2:v:63:false:1@3,])
> {code}
> Which on 3.0 side gets incorrectly deserialized as
> {code}
> Mutation(keyspace='distributed_test_keyspace', key='0001', modifications=[
> [distributed_test_keyspace.tbl] key=1
> partition_deletion=deletedAt=-9223372036854775808, localDeletion=2147483647
> columns=[[] | [v]]
> Row[info=[ts=-9223372036854775808] ]: ck=2 | del(v)=deletedAt=2,
> localDeletion=1588588821, [v[c]=d ts=3]
> Row[info=[ts=-9223372036854775808] del=deletedAt=1,
> localDeletion=1588588821 ]: ck=2 |
> Row[info=[ts=-9223372036854775808] ]: ck=3 | del(v)=deletedAt=2,
> localDeletion=1588588821
> ])
> {code}
> {{LegacyLayout}} correctly interprets a range tombstone whose start and
> finish {{collectionName}} values don't match as a wrapping fragment of a
> legacy row deletion that's being interrupted by a collection deletion
> (correctly) - see
> [code|https://github.com/apache/cassandra/blob/cassandra-3.0/src/java/org/apache/cassandra/db/LegacyLayout.java#L1874-L1889].
> Quoting the comment inline:
> {code}
> // Because of the way RangeTombstoneList work, we can have a tombstone where
> only one of
> // the bound has a collectionName. That happens if we have a big tombstone A
> (spanning one
> // or multiple rows) and a collection tombstone B. In that case,
> RangeTombstoneList will
> // split this into 3 RTs: the first one from the beginning of A to the
> beginning of B,
> // then B, then a third one from the end of B to the end of A. To make this
> simpler, if
> // we detect that case we transform the 1st and 3rd tombstone so they don't
> end in the middle
> // of a row (which is still correct).
>
[jira] [Updated] (CASSANDRA-15408) Cassandra throws SyntaxException for obsolete keywords that Thrift API permits
[
https://issues.apache.org/jira/browse/CASSANDRA-15408?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Leon Zaruvinsky updated CASSANDRA-15408:
Impacts: Docs (was: None)
Test and Documentation Plan: None; just a docs PR
Status: Patch Available (was: Open)
Added a quick patch for the docs change!
> Cassandra throws SyntaxException for obsolete keywords that Thrift API permits
> --
>
> Key: CASSANDRA-15408
> URL: https://issues.apache.org/jira/browse/CASSANDRA-15408
> Project: Cassandra
> Issue Type: Bug
> Components: Documentation/NEWS.txt
>Reporter: Leon Zaruvinsky
>Priority: Normal
> Fix For: 3.0.x, 3.11.x
>
> Attachments: CASSANDRA-15408.patch
>
>
> In [this
> refactor|https://github.com/apache/cassandra/commit/b31845c4a7982358a7c5bfd9bcf572fda6c1bfa9#diff-826a67bf1ae2e45372a35a6a2a6f3f3cL74]
> of CFPropDefs to TableAttributes for CASSANDRA-9712, three obsolete keywords
> were removed:
> {code:java}
> obsoleteKeywords.add("index_interval");
> obsoleteKeywords.add("replicate_on_write");
> obsoleteKeywords.add("populate_io_cache_on_flush");
> {code}
>
> The Thrift API continues to reference these keywords as deprecated, so it's
> not clear that they are actually unsupported.
> Could we either add them back as obsoleteKeywords, or add a change log that
> statements with these properties will fail (There is already a changelog
> about "index_interval" but not the other two)? I understand that the Thrift
> API is totally deprecated so I don't feel strongly about cleaning it up.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Updated] (CASSANDRA-15408) Cassandra throws SyntaxException for obsolete keywords that Thrift API permits
[
https://issues.apache.org/jira/browse/CASSANDRA-15408?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Leon Zaruvinsky updated CASSANDRA-15408:
Attachment: CASSANDRA-15408.patch
> Cassandra throws SyntaxException for obsolete keywords that Thrift API permits
> --
>
> Key: CASSANDRA-15408
> URL: https://issues.apache.org/jira/browse/CASSANDRA-15408
> Project: Cassandra
> Issue Type: Bug
> Components: Documentation/NEWS.txt
>Reporter: Leon Zaruvinsky
>Priority: Normal
> Fix For: 3.0.x, 3.11.x
>
> Attachments: CASSANDRA-15408.patch
>
>
> In [this
> refactor|https://github.com/apache/cassandra/commit/b31845c4a7982358a7c5bfd9bcf572fda6c1bfa9#diff-826a67bf1ae2e45372a35a6a2a6f3f3cL74]
> of CFPropDefs to TableAttributes for CASSANDRA-9712, three obsolete keywords
> were removed:
> {code:java}
> obsoleteKeywords.add("index_interval");
> obsoleteKeywords.add("replicate_on_write");
> obsoleteKeywords.add("populate_io_cache_on_flush");
> {code}
>
> The Thrift API continues to reference these keywords as deprecated, so it's
> not clear that they are actually unsupported.
> Could we either add them back as obsoleteKeywords, or add a change log that
> statements with these properties will fail (There is already a changelog
> about "index_interval" but not the other two)? I understand that the Thrift
> API is totally deprecated so I don't feel strongly about cleaning it up.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Updated] (CASSANDRA-15408) Cassandra throws SyntaxException for obsolete keywords that Thrift API permits
[
https://issues.apache.org/jira/browse/CASSANDRA-15408?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Leon Zaruvinsky updated CASSANDRA-15408:
Attachment: (was: CASSANDRA-15408.patch)
> Cassandra throws SyntaxException for obsolete keywords that Thrift API permits
> --
>
> Key: CASSANDRA-15408
> URL: https://issues.apache.org/jira/browse/CASSANDRA-15408
> Project: Cassandra
> Issue Type: Bug
> Components: Documentation/NEWS.txt
>Reporter: Leon Zaruvinsky
>Priority: Normal
> Fix For: 3.0.x, 3.11.x
>
> Attachments: CASSANDRA-15408.patch
>
>
> In [this
> refactor|https://github.com/apache/cassandra/commit/b31845c4a7982358a7c5bfd9bcf572fda6c1bfa9#diff-826a67bf1ae2e45372a35a6a2a6f3f3cL74]
> of CFPropDefs to TableAttributes for CASSANDRA-9712, three obsolete keywords
> were removed:
> {code:java}
> obsoleteKeywords.add("index_interval");
> obsoleteKeywords.add("replicate_on_write");
> obsoleteKeywords.add("populate_io_cache_on_flush");
> {code}
>
> The Thrift API continues to reference these keywords as deprecated, so it's
> not clear that they are actually unsupported.
> Could we either add them back as obsoleteKeywords, or add a change log that
> statements with these properties will fail (There is already a changelog
> about "index_interval" but not the other two)? I understand that the Thrift
> API is totally deprecated so I don't feel strongly about cleaning it up.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Commented] (CASSANDRA-15408) Cassandra throws SyntaxException for obsolete keywords that Thrift API permits
[
https://issues.apache.org/jira/browse/CASSANDRA-15408?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16971682#comment-16971682
]
Leon Zaruvinsky commented on CASSANDRA-15408:
-
Hey Aleksey - What are your thoughts on just adding a retroactive change log
signifying this break? While these keywords are no longer used, this is still
an API break in 3.x which we had to deal with during a migration.
> Cassandra throws SyntaxException for obsolete keywords that Thrift API permits
> --
>
> Key: CASSANDRA-15408
> URL: https://issues.apache.org/jira/browse/CASSANDRA-15408
> Project: Cassandra
> Issue Type: Bug
>Reporter: Leon Zaruvinsky
>Priority: Normal
> Attachments: CASSANDRA-15408.patch
>
>
> In [this
> refactor|https://github.com/apache/cassandra/commit/b31845c4a7982358a7c5bfd9bcf572fda6c1bfa9#diff-826a67bf1ae2e45372a35a6a2a6f3f3cL74]
> of CFPropDefs to TableAttributes for CASSANDRA-9712, three obsolete keywords
> were removed:
> {code:java}
> obsoleteKeywords.add("index_interval");
> obsoleteKeywords.add("replicate_on_write");
> obsoleteKeywords.add("populate_io_cache_on_flush");
> {code}
>
> The Thrift API continues to reference these keywords as deprecated, so it's
> not clear that they are actually unsupported.
> Could we either add them back as obsoleteKeywords, or add a change log that
> statements with these properties will fail (There is already a changelog
> about "index_interval" but not the other two)? I understand that the Thrift
> API is totally deprecated so I don't feel strongly about cleaning it up.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Updated] (CASSANDRA-15408) Cassandra throws SyntaxException for obsolete keywords that Thrift API permits
[
https://issues.apache.org/jira/browse/CASSANDRA-15408?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Leon Zaruvinsky updated CASSANDRA-15408:
Description:
In [this
refactor|https://github.com/apache/cassandra/commit/b31845c4a7982358a7c5bfd9bcf572fda6c1bfa9#diff-826a67bf1ae2e45372a35a6a2a6f3f3cL74]
of CFPropDefs to TableAttributes for CASSANDRA-9712, three obsolete keywords
were removed:
{code:java}
obsoleteKeywords.add("index_interval");
obsoleteKeywords.add("replicate_on_write");
obsoleteKeywords.add("populate_io_cache_on_flush");
{code}
The Thrift API continues to reference these keywords as deprecated, so it's not
clear that they are actually unsupported.
Could we either add them back as obsoleteKeywords, or add a change log that
statements with these properties will fail (There is already a changelog about
"index_interval" but not the other two)? I understand that the Thrift API is
totally deprecated so I don't feel strongly about cleaning it up.
was:
In [this
refactor|https://github.com/apache/cassandra/commit/b31845c4a7982358a7c5bfd9bcf572fda6c1bfa9#diff-826a67bf1ae2e45372a35a6a2a6f3f3cL74]
of CFPropDefs to TableAttributes for CASSANDRA-9712, three obsolete keywords
were removed:
{code:java}
obsoleteKeywords.add("index_interval");
obsoleteKeywords.add("replicate_on_write");
obsoleteKeywords.add("populate_io_cache_on_flush");
{code}
The Thrift API continues to reference these keywords as deprecated, so it's not
clear that they are actually unsupported.
Could we either add them back as obsoleteKeywords, or add a change log that
statements with these properties will fail? I understand that the Thrift API
is totally deprecated so I don't feel strongly about cleaning it up.
> Cassandra throws SyntaxException for obsolete keywords that Thrift API permits
> --
>
> Key: CASSANDRA-15408
> URL: https://issues.apache.org/jira/browse/CASSANDRA-15408
> Project: Cassandra
> Issue Type: Bug
>Reporter: Leon Zaruvinsky
>Priority: Normal
> Attachments: CASSANDRA-15408.patch
>
>
> In [this
> refactor|https://github.com/apache/cassandra/commit/b31845c4a7982358a7c5bfd9bcf572fda6c1bfa9#diff-826a67bf1ae2e45372a35a6a2a6f3f3cL74]
> of CFPropDefs to TableAttributes for CASSANDRA-9712, three obsolete keywords
> were removed:
> {code:java}
> obsoleteKeywords.add("index_interval");
> obsoleteKeywords.add("replicate_on_write");
> obsoleteKeywords.add("populate_io_cache_on_flush");
> {code}
>
> The Thrift API continues to reference these keywords as deprecated, so it's
> not clear that they are actually unsupported.
> Could we either add them back as obsoleteKeywords, or add a change log that
> statements with these properties will fail (There is already a changelog
> about "index_interval" but not the other two)? I understand that the Thrift
> API is totally deprecated so I don't feel strongly about cleaning it up.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Updated] (CASSANDRA-15408) Cassandra throws SyntaxException for obsolete keywords that Thrift API permits
[
https://issues.apache.org/jira/browse/CASSANDRA-15408?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Leon Zaruvinsky updated CASSANDRA-15408:
Attachment: CASSANDRA-15408.patch
> Cassandra throws SyntaxException for obsolete keywords that Thrift API permits
> --
>
> Key: CASSANDRA-15408
> URL: https://issues.apache.org/jira/browse/CASSANDRA-15408
> Project: Cassandra
> Issue Type: Bug
>Reporter: Leon Zaruvinsky
>Priority: Normal
> Attachments: CASSANDRA-15408.patch
>
>
> In [this
> refactor|https://github.com/apache/cassandra/commit/b31845c4a7982358a7c5bfd9bcf572fda6c1bfa9#diff-826a67bf1ae2e45372a35a6a2a6f3f3cL74]
> of CFPropDefs to TableAttributes for CASSANDRA-9712, three obsolete keywords
> were removed:
> {code:java}
> obsoleteKeywords.add("index_interval");
> obsoleteKeywords.add("replicate_on_write");
> obsoleteKeywords.add("populate_io_cache_on_flush");
> {code}
>
> The Thrift API continues to reference these keywords as deprecated, so it's
> not clear that they are actually unsupported.
> Could we either add them back as obsoleteKeywords, or add a change log that
> statements with these properties will fail? I understand that the Thrift API
> is totally deprecated so I don't feel strongly about cleaning it up.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Updated] (CASSANDRA-15408) Cassandra throws SyntaxException for obsolete keywords that Thrift API permits
[
https://issues.apache.org/jira/browse/CASSANDRA-15408?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Leon Zaruvinsky updated CASSANDRA-15408:
Description:
In [this
refactor|https://github.com/apache/cassandra/commit/b31845c4a7982358a7c5bfd9bcf572fda6c1bfa9#diff-826a67bf1ae2e45372a35a6a2a6f3f3cL74]
of CFPropDefs to TableAttributes for CASSANDRA-9712, three obsolete keywords
were removed:
{code:java}
obsoleteKeywords.add("index_interval");
obsoleteKeywords.add("replicate_on_write");
obsoleteKeywords.add("populate_io_cache_on_flush");
{code}
The Thrift API continues to reference these keywords as deprecated, so it's not
clear that they are actually unsupported.
Could we either add them back as obsoleteKeywords, or add a change log that
statements with these properties will fail? I understand that the Thrift API
is totally deprecated so I don't feel strongly about cleaning it up.
was:
In [this
refactor|https://github.com/apache/cassandra/commit/b31845c4a7982358a7c5bfd9bcf572fda6c1bfa9#diff-826a67bf1ae2e45372a35a6a2a6f3f3cL74]
of CFPropDefs to TableAttributes for CASSANDRA-9712, three obsolete keywords
were removed:
{code:java}
obsoleteKeywords.add("index_interval");
obsoleteKeywords.add("replicate_on_write");
obsoleteKeywords.add("populate_io_cache_on_flush");
{code}
The Thrift API continues to reference these keywords as deprecated, so it's not
clear that they are actually unsupported.
Could we either add them back as obsoleteKeywords, or remove them from the
Thrift API and add a change log that statements with these properties will fail?
> Cassandra throws SyntaxException for obsolete keywords that Thrift API permits
> --
>
> Key: CASSANDRA-15408
> URL: https://issues.apache.org/jira/browse/CASSANDRA-15408
> Project: Cassandra
> Issue Type: Bug
>Reporter: Leon Zaruvinsky
>Priority: Normal
>
> In [this
> refactor|https://github.com/apache/cassandra/commit/b31845c4a7982358a7c5bfd9bcf572fda6c1bfa9#diff-826a67bf1ae2e45372a35a6a2a6f3f3cL74]
> of CFPropDefs to TableAttributes for CASSANDRA-9712, three obsolete keywords
> were removed:
> {code:java}
> obsoleteKeywords.add("index_interval");
> obsoleteKeywords.add("replicate_on_write");
> obsoleteKeywords.add("populate_io_cache_on_flush");
> {code}
>
> The Thrift API continues to reference these keywords as deprecated, so it's
> not clear that they are actually unsupported.
> Could we either add them back as obsoleteKeywords, or add a change log that
> statements with these properties will fail? I understand that the Thrift API
> is totally deprecated so I don't feel strongly about cleaning it up.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Updated] (CASSANDRA-15408) Cassandra throws SyntaxException for obsolete keywords that Thrift API permits
[
https://issues.apache.org/jira/browse/CASSANDRA-15408?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Leon Zaruvinsky updated CASSANDRA-15408:
Description:
In [this
refactor|https://github.com/apache/cassandra/commit/b31845c4a7982358a7c5bfd9bcf572fda6c1bfa9#diff-826a67bf1ae2e45372a35a6a2a6f3f3cL74]
of CFPropDefs to TableAttributes for CASSANDRA-9712, three obsolete keywords
were removed:
{code:java}
obsoleteKeywords.add("index_interval");
obsoleteKeywords.add("replicate_on_write");
obsoleteKeywords.add("populate_io_cache_on_flush");
{code}
The Thrift API continues to reference these keywords as deprecated, so it's not
clear that they are actually unsupported.
Could we either add them back as obsoleteKeywords, or remove them from the
Thrift API and add a change log that statements with these properties will fail?
was:
In [this refactor|#diff-826a67bf1ae2e45372a35a6a2a6f3f3cL74] of CFPropDefs to
TableAttributes, three obsolete keywords were removed:
{code:java}
obsoleteKeywords.add("index_interval");
obsoleteKeywords.add("replicate_on_write");
obsoleteKeywords.add("populate_io_cache_on_flush");
{code}
The Thrift API continues to reference these keywords as deprecated, so it's not
clear that they are actually unsupported.
Could we either add them back as obsoleteKeywords, or remove them from the
Thrift API and add a change log that statements with these properties will fail?
> Cassandra throws SyntaxException for obsolete keywords that Thrift API permits
> --
>
> Key: CASSANDRA-15408
> URL: https://issues.apache.org/jira/browse/CASSANDRA-15408
> Project: Cassandra
> Issue Type: Bug
>Reporter: Leon Zaruvinsky
>Priority: Normal
>
> In [this
> refactor|https://github.com/apache/cassandra/commit/b31845c4a7982358a7c5bfd9bcf572fda6c1bfa9#diff-826a67bf1ae2e45372a35a6a2a6f3f3cL74]
> of CFPropDefs to TableAttributes for CASSANDRA-9712, three obsolete keywords
> were removed:
> {code:java}
> obsoleteKeywords.add("index_interval");
> obsoleteKeywords.add("replicate_on_write");
> obsoleteKeywords.add("populate_io_cache_on_flush");
> {code}
>
> The Thrift API continues to reference these keywords as deprecated, so it's
> not clear that they are actually unsupported.
> Could we either add them back as obsoleteKeywords, or remove them from the
> Thrift API and add a change log that statements with these properties will
> fail?
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Updated] (CASSANDRA-15408) Cassandra throws SyntaxException for obsolete keywords that Thrift API permits
[
https://issues.apache.org/jira/browse/CASSANDRA-15408?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Leon Zaruvinsky updated CASSANDRA-15408:
Description:
In [this refactor|#diff-826a67bf1ae2e45372a35a6a2a6f3f3cL74] of CFPropDefs to
TableAttributes, three obsolete keywords were removed:
{code:java}
obsoleteKeywords.add("index_interval");
obsoleteKeywords.add("replicate_on_write");
obsoleteKeywords.add("populate_io_cache_on_flush");
{code}
The Thrift API continues to reference these keywords as deprecated, so it's not
clear that they are actually unsupported.
Could we either add them back as obsoleteKeywords, or remove them from the
Thrift API and add a change log that statements with these properties will fail?
was:
In [this refactor|#diff-826a67bf1ae2e45372a35a6a2a6f3f3cL74]] of CFPropDefs to
TableAttributes, three obsolete keywords were removed:
{code:java}
obsoleteKeywords.add("index_interval");
obsoleteKeywords.add("replicate_on_write");
obsoleteKeywords.add("populate_io_cache_on_flush");
{code}
The Thrift API continues to reference these keywords as deprecated, so it's not
clear that they are actually unsupported.
Could we either add them back as obsoleteKeywords, or remove them from the
Thrift API and add a change log that statements with these properties will fail?
> Cassandra throws SyntaxException for obsolete keywords that Thrift API permits
> --
>
> Key: CASSANDRA-15408
> URL: https://issues.apache.org/jira/browse/CASSANDRA-15408
> Project: Cassandra
> Issue Type: Bug
>Reporter: Leon Zaruvinsky
>Priority: Normal
>
> In [this refactor|#diff-826a67bf1ae2e45372a35a6a2a6f3f3cL74] of CFPropDefs to
> TableAttributes, three obsolete keywords were removed:
> {code:java}
> obsoleteKeywords.add("index_interval");
> obsoleteKeywords.add("replicate_on_write");
> obsoleteKeywords.add("populate_io_cache_on_flush");
> {code}
>
> The Thrift API continues to reference these keywords as deprecated, so it's
> not clear that they are actually unsupported.
> Could we either add them back as obsoleteKeywords, or remove them from the
> Thrift API and add a change log that statements with these properties will
> fail?
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Created] (CASSANDRA-15408) Cassandra throws SyntaxException for obsolete keywords that Thrift API permits
Leon Zaruvinsky created CASSANDRA-15408:
---
Summary: Cassandra throws SyntaxException for obsolete keywords
that Thrift API permits
Key: CASSANDRA-15408
URL: https://issues.apache.org/jira/browse/CASSANDRA-15408
Project: Cassandra
Issue Type: Bug
Reporter: Leon Zaruvinsky
In [this refactor|#diff-826a67bf1ae2e45372a35a6a2a6f3f3cL74]] of CFPropDefs to
TableAttributes, three obsolete keywords were removed:
{code:java}
obsoleteKeywords.add("index_interval");
obsoleteKeywords.add("replicate_on_write");
obsoleteKeywords.add("populate_io_cache_on_flush");
{code}
The Thrift API continues to reference these keywords as deprecated, so it's not
clear that they are actually unsupported.
Could we either add them back as obsoleteKeywords, or remove them from the
Thrift API and add a change log that statements with these properties will fail?
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Commented] (CASSANDRA-14781) Log message when mutation passed to CommitLog#add(Mutation) is too large is not descriptive enough
[ https://issues.apache.org/jira/browse/CASSANDRA-14781?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16944747#comment-16944747 ] Leon Zaruvinsky commented on CASSANDRA-14781: - [^CASSANDRA-14781_3.11.patch] Hey [~cnlwsu] [~jrwest], I've attached a patch for 3.11 (should be the same for 3.0) that prints (cf, key) pairs with a limit of 10. > Log message when mutation passed to CommitLog#add(Mutation) is too large is > not descriptive enough > -- > > Key: CASSANDRA-14781 > URL: https://issues.apache.org/jira/browse/CASSANDRA-14781 > Project: Cassandra > Issue Type: Bug > Components: Legacy/Local Write-Read Paths >Reporter: Jordan West >Assignee: Tom Petracca >Priority: Normal > Fix For: 3.0.x, 3.11.x, 4.0.x > > Attachments: CASSANDRA-14781.patch, CASSANDRA-14781_3.0.patch, > CASSANDRA-14781_3.11.patch > > > When hitting > [https://github.com/apache/cassandra/blob/cassandra-3.0/src/java/org/apache/cassandra/db/commitlog/CommitLog.java#L256-L257], > the log message produced does not help the operator track down what data is > being written. At a minimum the keyspace and cfIds involved would be useful > (and are available) – more detail might not be reasonable to include. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Updated] (CASSANDRA-14781) Log message when mutation passed to CommitLog#add(Mutation) is too large is not descriptive enough
[ https://issues.apache.org/jira/browse/CASSANDRA-14781?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Leon Zaruvinsky updated CASSANDRA-14781: Attachment: CASSANDRA-14781_3.11.patch > Log message when mutation passed to CommitLog#add(Mutation) is too large is > not descriptive enough > -- > > Key: CASSANDRA-14781 > URL: https://issues.apache.org/jira/browse/CASSANDRA-14781 > Project: Cassandra > Issue Type: Bug > Components: Legacy/Local Write-Read Paths >Reporter: Jordan West >Assignee: Tom Petracca >Priority: Normal > Fix For: 3.0.x, 3.11.x, 4.0.x > > Attachments: CASSANDRA-14781.patch, CASSANDRA-14781_3.0.patch, > CASSANDRA-14781_3.11.patch > > > When hitting > [https://github.com/apache/cassandra/blob/cassandra-3.0/src/java/org/apache/cassandra/db/commitlog/CommitLog.java#L256-L257], > the log message produced does not help the operator track down what data is > being written. At a minimum the keyspace and cfIds involved would be useful > (and are available) – more detail might not be reasonable to include. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Updated] (CASSANDRA-15327) Deleted data can re-appear if range movement streaming time exceeds gc_grace_seconds
[
https://issues.apache.org/jira/browse/CASSANDRA-15327?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Leon Zaruvinsky updated CASSANDRA-15327:
Discovered By: (was: User Report)
> Deleted data can re-appear if range movement streaming time exceeds
> gc_grace_seconds
>
>
> Key: CASSANDRA-15327
> URL: https://issues.apache.org/jira/browse/CASSANDRA-15327
> Project: Cassandra
> Issue Type: Bug
> Components: Consistency/Bootstrap and Decommission
>Reporter: Leon Zaruvinsky
>Priority: Normal
> Fix For: 2.2.15, 2.1.x
>
> Attachments: CASSANDRA-15327-2.1.txt, CASSANDRA-15327-2.2.txt
>
>
> Hey,
> We've come across a scenario in production (noticed on Cassandra 2.2.14)
> where data that is deleted from Cassandra at consistency {{ALL}} can be
> resurrected. I've added a reproduction in a comment.
> If a {{delete}} is issued during a range movement (i.e. bootstrap,
> decommission, move), and {{gc_grace_seconds}} is surpassed before the stream
> is finished, then the tombstones from the {{delete}} can be purged from the
> recipient node before the data is streamed. Once the move is complete, the
> data now exists on the recipient node without a tombstone.
> We noticed this because our bootstrapping time occasionally exceeds our
> configured gc_grace_seconds, so we lose the consistency guarantee. As an
> operator, it would be great to not have to worry about this edge case.
> I've attached a patch that we have tested and successfully used in
> production, and haven't noticed any ill effects. Happy to submit patches for
> more recent versions, I'm not sure how cleanly this will actually merge since
> there was some refactoring to this logic in 3.x.
--
This message was sent by Atlassian Jira
(v8.3.2#803003)
-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Updated] (CASSANDRA-15327) Deleted data can re-appear if range movement streaming time exceeds gc_grace_seconds
[
https://issues.apache.org/jira/browse/CASSANDRA-15327?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Leon Zaruvinsky updated CASSANDRA-15327:
Description:
Hey,
We've come across a scenario in production (noticed on Cassandra 2.2.14) where
data that is deleted from Cassandra at consistency {{ALL}} can be resurrected.
I've added a reproduction in a comment.
If a {{delete}} is issued during a range movement (i.e. bootstrap,
decommission, move), and {{gc_grace_seconds}} is surpassed before the stream is
finished, then the tombstones from the {{delete}} can be purged from the
recipient node before the data is streamed. Once the move is complete, the data
now exists on the recipient node without a tombstone.
We noticed this because our bootstrapping time occasionally exceeds our
configured gc_grace_seconds, so we lose the consistency guarantee. As an
operator, it would be great to not have to worry about this edge case.
I've attached a patch that we have tested and successfully used in production,
and haven't noticed any ill effects. Happy to submit patches for more recent
versions, I'm not sure how cleanly this will actually merge since there was
some refactoring to this logic in 3.x.
was:
Hey,
We've come across a scenario in production (noticed on Cassandra 2.2.14) where
data that is deleted from Cassandra at consistency {{ALL}} can be resurrected.
I've added a reproduction in a comment.
If a {{delete}} is issued during a range movement (i.e. bootstrap,
decommission, move), and {{gc_grace_seconds}} is surpassed before the stream is
finished, then the tombstones from the {{delete}} can be purged from the
recipient node before the data is streamed. Once the move is complete, the data
now exists on the recipient node without a tombstone.
We noticed this because our bootstrapping time occasionally exceeds our
configured gc_grace_seconds, so we lose the consistency guarantee. As an
operator, it would be great to not have to worry about this edge case.
I've attached a patch that we have tested and successfully used in production,
and haven't noticed any ill effects.
> Deleted data can re-appear if range movement streaming time exceeds
> gc_grace_seconds
>
>
> Key: CASSANDRA-15327
> URL: https://issues.apache.org/jira/browse/CASSANDRA-15327
> Project: Cassandra
> Issue Type: Bug
> Components: Consistency/Bootstrap and Decommission
>Reporter: Leon Zaruvinsky
>Priority: Normal
> Fix For: 2.2.15, 2.1.x
>
> Attachments: CASSANDRA-15327-2.1.txt, CASSANDRA-15327-2.2.txt
>
>
> Hey,
> We've come across a scenario in production (noticed on Cassandra 2.2.14)
> where data that is deleted from Cassandra at consistency {{ALL}} can be
> resurrected. I've added a reproduction in a comment.
> If a {{delete}} is issued during a range movement (i.e. bootstrap,
> decommission, move), and {{gc_grace_seconds}} is surpassed before the stream
> is finished, then the tombstones from the {{delete}} can be purged from the
> recipient node before the data is streamed. Once the move is complete, the
> data now exists on the recipient node without a tombstone.
> We noticed this because our bootstrapping time occasionally exceeds our
> configured gc_grace_seconds, so we lose the consistency guarantee. As an
> operator, it would be great to not have to worry about this edge case.
> I've attached a patch that we have tested and successfully used in
> production, and haven't noticed any ill effects. Happy to submit patches for
> more recent versions, I'm not sure how cleanly this will actually merge since
> there was some refactoring to this logic in 3.x.
--
This message was sent by Atlassian Jira
(v8.3.2#803003)
-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Updated] (CASSANDRA-15327) Deleted data can re-appear if range movement streaming time exceeds gc_grace_seconds
[
https://issues.apache.org/jira/browse/CASSANDRA-15327?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Leon Zaruvinsky updated CASSANDRA-15327:
Bug Category: Parent values: Correctness(12982)
Complexity: Normal
Discovered By: User Report
Fix Version/s: 2.1.x
2.2.15
Platform: (was: All)
Severity: Normal
Status: Open (was: Triage Needed)
> Deleted data can re-appear if range movement streaming time exceeds
> gc_grace_seconds
>
>
> Key: CASSANDRA-15327
> URL: https://issues.apache.org/jira/browse/CASSANDRA-15327
> Project: Cassandra
> Issue Type: Bug
> Components: Consistency/Bootstrap and Decommission
>Reporter: Leon Zaruvinsky
>Priority: Normal
> Fix For: 2.2.15, 2.1.x
>
> Attachments: CASSANDRA-15327-2.1.txt, CASSANDRA-15327-2.2.txt
>
>
> Hey,
> We've come across a scenario in production (noticed on Cassandra 2.2.14)
> where data that is deleted from Cassandra at consistency {{ALL}} can be
> resurrected. I've added a reproduction in a comment.
> If a {{delete}} is issued during a range movement (i.e. bootstrap,
> decommission, move), and {{gc_grace_seconds}} is surpassed before the stream
> is finished, then the tombstones from the {{delete}} can be purged from the
> recipient node before the data is streamed. Once the move is complete, the
> data now exists on the recipient node without a tombstone.
> We noticed this because our bootstrapping time occasionally exceeds our
> configured gc_grace_seconds, so we lose the consistency guarantee. As an
> operator, it would be great to not have to worry about this edge case.
> I've attached a patch that we have tested and successfully used in
> production, and haven't noticed any ill effects.
--
This message was sent by Atlassian Jira
(v8.3.2#803003)
-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Updated] (CASSANDRA-15327) Deleted data can re-appear if range movement streaming time exceeds gc_grace_seconds
[
https://issues.apache.org/jira/browse/CASSANDRA-15327?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Leon Zaruvinsky updated CASSANDRA-15327:
Attachment: CASSANDRA-15327-2.2.txt
> Deleted data can re-appear if range movement streaming time exceeds
> gc_grace_seconds
>
>
> Key: CASSANDRA-15327
> URL: https://issues.apache.org/jira/browse/CASSANDRA-15327
> Project: Cassandra
> Issue Type: Bug
> Components: Consistency/Bootstrap and Decommission
>Reporter: Leon Zaruvinsky
>Priority: Normal
> Attachments: CASSANDRA-15327-2.1.txt, CASSANDRA-15327-2.2.txt
>
>
> Hey,
> We've come across a scenario in production (noticed on Cassandra 2.2.14)
> where data that is deleted from Cassandra at consistency {{ALL}} can be
> resurrected. I've added a reproduction in a comment.
> If a {{delete}} is issued during a range movement (i.e. bootstrap,
> decommission, move), and {{gc_grace_seconds}} is surpassed before the stream
> is finished, then the tombstones from the {{delete}} can be purged from the
> recipient node before the data is streamed. Once the move is complete, the
> data now exists on the recipient node without a tombstone.
> We noticed this because our bootstrapping time occasionally exceeds our
> configured gc_grace_seconds, so we lose the consistency guarantee. As an
> operator, it would be great to not have to worry about this edge case.
> I've attached a patch that we have tested and successfully used in
> production, and haven't noticed any ill effects.
--
This message was sent by Atlassian Jira
(v8.3.2#803003)
-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Updated] (CASSANDRA-15327) Deleted data can re-appear if range movement streaming time exceeds gc_grace_seconds
[
https://issues.apache.org/jira/browse/CASSANDRA-15327?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Leon Zaruvinsky updated CASSANDRA-15327:
Attachment: CASSANDRA-15327-2.2.txt
> Deleted data can re-appear if range movement streaming time exceeds
> gc_grace_seconds
>
>
> Key: CASSANDRA-15327
> URL: https://issues.apache.org/jira/browse/CASSANDRA-15327
> Project: Cassandra
> Issue Type: Bug
> Components: Consistency/Bootstrap and Decommission
>Reporter: Leon Zaruvinsky
>Priority: Normal
> Attachments: CASSANDRA-15327-2.1.txt
>
>
> Hey,
> We've come across a scenario in production (noticed on Cassandra 2.2.14)
> where data that is deleted from Cassandra at consistency {{ALL}} can be
> resurrected. I've added a reproduction in a comment.
> If a {{delete}} is issued during a range movement (i.e. bootstrap,
> decommission, move), and {{gc_grace_seconds}} is surpassed before the stream
> is finished, then the tombstones from the {{delete}} can be purged from the
> recipient node before the data is streamed. Once the move is complete, the
> data now exists on the recipient node without a tombstone.
> We noticed this because our bootstrapping time occasionally exceeds our
> configured gc_grace_seconds, so we lose the consistency guarantee. As an
> operator, it would be great to not have to worry about this edge case.
> I've attached a patch that we have tested and successfully used in
> production, and haven't noticed any ill effects.
--
This message was sent by Atlassian Jira
(v8.3.2#803003)
-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Updated] (CASSANDRA-15327) Deleted data can re-appear if range movement streaming time exceeds gc_grace_seconds
[
https://issues.apache.org/jira/browse/CASSANDRA-15327?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Leon Zaruvinsky updated CASSANDRA-15327:
Attachment: (was: CASSANDRA-15327-2.2.txt)
> Deleted data can re-appear if range movement streaming time exceeds
> gc_grace_seconds
>
>
> Key: CASSANDRA-15327
> URL: https://issues.apache.org/jira/browse/CASSANDRA-15327
> Project: Cassandra
> Issue Type: Bug
> Components: Consistency/Bootstrap and Decommission
>Reporter: Leon Zaruvinsky
>Priority: Normal
> Attachments: CASSANDRA-15327-2.1.txt
>
>
> Hey,
> We've come across a scenario in production (noticed on Cassandra 2.2.14)
> where data that is deleted from Cassandra at consistency {{ALL}} can be
> resurrected. I've added a reproduction in a comment.
> If a {{delete}} is issued during a range movement (i.e. bootstrap,
> decommission, move), and {{gc_grace_seconds}} is surpassed before the stream
> is finished, then the tombstones from the {{delete}} can be purged from the
> recipient node before the data is streamed. Once the move is complete, the
> data now exists on the recipient node without a tombstone.
> We noticed this because our bootstrapping time occasionally exceeds our
> configured gc_grace_seconds, so we lose the consistency guarantee. As an
> operator, it would be great to not have to worry about this edge case.
> I've attached a patch that we have tested and successfully used in
> production, and haven't noticed any ill effects.
--
This message was sent by Atlassian Jira
(v8.3.2#803003)
-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Commented] (CASSANDRA-15327) Deleted data can re-appear if range movement streaming time exceeds gc_grace_seconds
[
https://issues.apache.org/jira/browse/CASSANDRA-15327?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16930940#comment-16930940
]
Leon Zaruvinsky commented on CASSANDRA-15327:
-
h2. Reproduction (with Bootstrap)
h3. Cluster setup
Apache Cassandra 2.2.14, 3 nodes with SimpleSeedProvider.
{code:java}
$ nodetool status
Datacenter: DC11
Status=Up/Down
|/ State=Normal/Leaving/Joining/Moving
-- Address Load Tokens Owns (effective) Host ID Rack
UN x.x.x.180 101.68 KB 512 66.7% 5b073424-1f21-4aac-acd5-e0e8e82f7073 RAC1
UN x.x.x.217 101.64 KB 512 67.6% 20e9ecd3-b3a5-4171-901c-eb6e7bc7a429 RAC1
UN x.x.x.142 101.69 KB 512 65.7% 1df2908e-5697-4b1a-9110-dcf581511510 RAC1
{code}
h3. Seed Data
We create a table {{datadump.ddt}} using {{LeveledCompactionStrategy}}, with
{{RF = 3}} and {{gc_grace_seconds = 1}}. We write at {{QUORUM}} 20,000 rows of
data in the format (row, value) where row is a partition key [0, 2) and
value is a one megabyte blob.
{code:java}
cqlsh> describe datadump;
CREATE KEYSPACE datadump WITH replication = {'class': 'SimpleStrategy',
'replication_factor': '3'} AND durable_writes = true;
CREATE TABLE datadump.ddt (
row bigint,
col bigint,
value blob static,
PRIMARY KEY (row, col)
) WITH CLUSTERING ORDER BY (col ASC)
AND bloom_filter_fp_chance = 0.1
AND caching = '{"keys":"ALL", "rows_per_partition":"NONE"}'
AND comment = ''
AND compaction = {'class':
'org.apache.cassandra.db.compaction.LeveledCompactionStrategy'}
AND compression = {}
AND dclocal_read_repair_chance = 0.1
AND default_time_to_live = 0
AND gc_grace_seconds = 1
AND max_index_interval = 2048
AND memtable_flush_period_in_ms = 0
AND min_index_interval = 128
AND read_repair_chance = 0.0
AND speculative_retry = '99.0PERCENTILE';
cqlsh> SELECT COUNT(row) from datadump.ddt;
system.count(row)
2
$ nodetool status
Datacenter: DC11
Status=Up/Down
|/ State=Normal/Leaving/Joining/Moving
-- Address Load Tokens Owns (effective) Host ID Rack
UN x.x.x.180 19.26 GB 512 100.0% 5b073424-1f21-4aac-acd5-e0e8e82f7073 RAC1
UN x.x.x.217 19.26 GB 512 100.0% 20e9ecd3-b3a5-4171-901c-eb6e7bc7a429 RAC1
UN x.x.x.142 18.71 GB 512 100.0% 1df2908e-5697-4b1a-9110-dcf581511510 RAC1
{code}
h3. Bootstrap + Delete
We bootstrap a fourth node to the cluster, and as soon as it begins to stream
data, we delete at {{ALL}} the 20,000 rows that we inserted earlier. The
deletes will be forwarded to the bootstrapping node immediately. As soon as the
delete queries complete, we run a flush and major compaction on the
bootstrapping node to clear the tombstones.
{code:java}
$ nodetool status
Datacenter: DC11
Status=Up/Down
|/ State=Normal/Leaving/Joining/Moving
-- Address Load Tokens Owns (effective) Host ID Rack
UJ x.x.x.129 14.43 KB 512 ? 8418418b-2aa2-4918-a065-8fc25887194f RAC1
UN x.x.x.180 19.26 GB 512 100.0% 5b073424-1f21-4aac-acd5-e0e8e82f7073 RAC1
UN x.x.x.217 19.26 GB 512 100.0% 20e9ecd3-b3a5-4171-901c-eb6e7bc7a429 RAC1
UN x.x.x.142 18.71 GB 512 100.0% 1df2908e-5697-4b1a-9110-dcf581511510 RAC1
// First, we trigger deletes on one of the original three nodes.
// Then, we run the below on the bootstrapping node
$ nodetool flush && nodetool compact
$ nodetool compactionhistory
{code}
Compaction History:
{code:java}
id keyspace_name columnfamily_name compacted_at bytes_in bytes_out rows_merged
b63e7e30-d00a-11e9-a7b9-f18b3a65a899 datadump ddt 1567708003731 366768 0 {}
7e2253a0-d00a-11e9-a7b9-f18b3a65a899 system local 1567707909594 10705 10500
{4:1}
{code}
h3. Final State
Once the bootstrap has completed, the cluster looks like this:
{code:java}
$ nodetool status
Datacenter: DC11
Status=Up/Down
|/ State=Normal/Leaving/Joining/Moving
-- Address Load Tokens Owns (effective) Host ID Rack
UN x.x.x.129 14.93 GB 512 76.1% 8418418b-2aa2-4918-a065-8fc25887194f RAC1
UN x.x.x.180 19.54 GB 512 74.5% 5b073424-1f21-4aac-acd5-e0e8e82f7073 RAC1
UN x.x.x.217 19.54 GB 512 75.3% 20e9ecd3-b3a5-4171-901c-eb6e7bc7a429 RAC1
UN x.x.x.142 19.54 GB 512 74.1% 1df2908e-5697-4b1a-9110-dcf581511510 RAC1{code}
We run a flush and major compaction on every node. The original three nodes
drop everything, but the bootstrapped node holds onto nearly 75% of the data.
{code:java}
$ nodetool status
Datacenter: DC11
Status=Up/Down
|/ State=Normal/Leaving/Joining/Moving
-- Address Load Tokens Owns (effective) Host ID Rack
UN x.x.x.129 14.93 GB 512 76.1% 8418418b-2aa2-4918-a065-8fc25887194f RAC1
UN x.x.x.180 143.39 KB 512 74.5% 5b073424-1f21-4aac-acd5-e0e8e82f7073 RAC1
UN x.x.x.217 128.6 KB 512 75.3% 20e9ecd3-b3a5-4171-901c-eb6e7bc7a429 RAC1
UN x.x.x.142 149.97 KB 512 74.1% 1df2908e-5697-4b1a-9110-dcf581511510 RAC1{code}
If we run the count query again, we see the data get read-repaired back into
the other nodes.
{code:java}
$ nodetool status
Datacenter: DC11
Status=Up/Down
|/ State=Normal/Leaving/Joining/Moving
-- Address Load Tokens Owns (effective) Host ID Rack
UN x.x.x.129
[jira] [Updated] (CASSANDRA-15327) Deleted data can re-appear if range movement streaming time exceeds gc_grace_seconds
[
https://issues.apache.org/jira/browse/CASSANDRA-15327?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Leon Zaruvinsky updated CASSANDRA-15327:
Description:
Hey,
We've come across a scenario in production (noticed on Cassandra 2.2.14) where
data that is deleted from Cassandra at consistency {{ALL}} can be resurrected.
I've added a reproduction in a comment.
If a {{delete}} is issued during a range movement (i.e. bootstrap,
decommission, move), and {{gc_grace_seconds}} is surpassed before the stream is
finished, then the tombstones from the {{delete}} can be purged from the
recipient node before the data is streamed. Once the move is complete, the data
now exists on the recipient node without a tombstone.
We noticed this because our bootstrapping time occasionally exceeds our
configured gc_grace_seconds, so we lose the consistency guarantee. As an
operator, it would be great to not have to worry about this edge case.
I've attached a patch that we have tested and successfully used in production,
and haven't noticed any ill effects.
was:
Hey,
We've come across a scenario in production (noticed on Cassandra 2.2.14) where
data that is deleted from Cassandra at consistency {{ALL}} can be resurrected.
I've added a reproduction in a comment.
If a {{delete}} is issued during a range movement (i.e. bootstrap,
decommission, move), and {{gc_grace_seconds}} is surpassed before the stream is
finished, then the tombstones from the {{delete}} can be purged from the
recipient node before the data is streamed. Once the move is complete, the data
now exists on the recipient node without a tombstone.
We noticed this because our bootstrapping time occasionally exceeds our
configured gc_grace_seconds, so we lose the consistency guarantee. As an
operator, it would be great to not have to worry about this edge case.
I've attached a patch that we have tested and successfully used in production,
and haven't noticed any ill effects.
> Deleted data can re-appear if range movement streaming time exceeds
> gc_grace_seconds
>
>
> Key: CASSANDRA-15327
> URL: https://issues.apache.org/jira/browse/CASSANDRA-15327
> Project: Cassandra
> Issue Type: Bug
> Components: Consistency/Bootstrap and Decommission
>Reporter: Leon Zaruvinsky
>Priority: Normal
> Attachments: CASSANDRA-15327-2.1.txt
>
>
> Hey,
> We've come across a scenario in production (noticed on Cassandra 2.2.14)
> where data that is deleted from Cassandra at consistency {{ALL}} can be
> resurrected. I've added a reproduction in a comment.
> If a {{delete}} is issued during a range movement (i.e. bootstrap,
> decommission, move), and {{gc_grace_seconds}} is surpassed before the stream
> is finished, then the tombstones from the {{delete}} can be purged from the
> recipient node before the data is streamed. Once the move is complete, the
> data now exists on the recipient node without a tombstone.
> We noticed this because our bootstrapping time occasionally exceeds our
> configured gc_grace_seconds, so we lose the consistency guarantee. As an
> operator, it would be great to not have to worry about this edge case.
> I've attached a patch that we have tested and successfully used in
> production, and haven't noticed any ill effects.
--
This message was sent by Atlassian Jira
(v8.3.2#803003)
-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Comment Edited] (CASSANDRA-15327) Deleted data can re-appear if range movement streaming time exceeds gc_grace_seconds
[
https://issues.apache.org/jira/browse/CASSANDRA-15327?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16930940#comment-16930940
]
Leon Zaruvinsky edited comment on CASSANDRA-15327 at 9/16/19 10:54 PM:
---
h2. Reproduction (with Bootstrap)
h3. Cluster setup
Apache Cassandra 2.2.14, 3 nodes with SimpleSeedProvider.
{code:java}
$ nodetool status
Datacenter: DC11
Status=Up/Down
|/ State=Normal/Leaving/Joining/Moving
-- Address Load Tokens Owns (effective) Host ID Rack
UN x.x.x.180 101.68 KB 512 66.7% 5b073424-1f21-4aac-acd5-e0e8e82f7073 RAC1
UN x.x.x.217 101.64 KB 512 67.6% 20e9ecd3-b3a5-4171-901c-eb6e7bc7a429 RAC1
UN x.x.x.142 101.69 KB 512 65.7% 1df2908e-5697-4b1a-9110-dcf581511510 RAC1
{code}
h3. Seed Data
We create a table {{datadump.ddt}} using {{LeveledCompactionStrategy}}, with
{{RF = 3}} and {{gc_grace_seconds = 1}}. We write at {{QUORUM}} 20,000 rows of
data in the format (row, value) where row is a partition key [0, 2) and
value is a one megabyte blob.
{code:java}
cqlsh> describe datadump;
CREATE KEYSPACE datadump WITH replication = {'class': 'SimpleStrategy',
'replication_factor': '3'} AND durable_writes = true;
CREATE TABLE datadump.ddt (
row bigint,
col bigint,
value blob static,
PRIMARY KEY (row, col)
) WITH CLUSTERING ORDER BY (col ASC)
AND bloom_filter_fp_chance = 0.1
AND caching = '{"keys":"ALL", "rows_per_partition":"NONE"}'
AND comment = ''
AND compaction = {'class':
'org.apache.cassandra.db.compaction.LeveledCompactionStrategy'}
AND compression = {}
AND dclocal_read_repair_chance = 0.1
AND default_time_to_live = 0
AND gc_grace_seconds = 1
AND max_index_interval = 2048
AND memtable_flush_period_in_ms = 0
AND min_index_interval = 128
AND read_repair_chance = 0.0
AND speculative_retry = '99.0PERCENTILE';
cqlsh> SELECT COUNT(row) from datadump.ddt;
system.count(row)
2
$ nodetool status
Datacenter: DC11
Status=Up/Down
|/ State=Normal/Leaving/Joining/Moving
-- Address Load Tokens Owns (effective) Host ID Rack
UN x.x.x.180 19.26 GB 512 100.0% 5b073424-1f21-4aac-acd5-e0e8e82f7073 RAC1
UN x.x.x.217 19.26 GB 512 100.0% 20e9ecd3-b3a5-4171-901c-eb6e7bc7a429 RAC1
UN x.x.x.142 18.71 GB 512 100.0% 1df2908e-5697-4b1a-9110-dcf581511510 RAC1
{code}
h3. Bootstrap + Delete
We bootstrap a fourth node to the cluster, and as soon as it begins to stream
data, we delete at {{ALL}} the 20,000 rows that we inserted earlier. The
deletes will be forwarded to the bootstrapping node immediately. As soon as the
delete queries complete, we run a flush and major compaction on the
bootstrapping node to clear the tombstones.
{code:java}
$ nodetool status
Datacenter: DC11
Status=Up/Down
|/ State=Normal/Leaving/Joining/Moving
-- Address Load Tokens Owns (effective) Host ID Rack
UJ x.x.x.129 14.43 KB 512 ? 8418418b-2aa2-4918-a065-8fc25887194f RAC1
UN x.x.x.180 19.26 GB 512 100.0% 5b073424-1f21-4aac-acd5-e0e8e82f7073 RAC1
UN x.x.x.217 19.26 GB 512 100.0% 20e9ecd3-b3a5-4171-901c-eb6e7bc7a429 RAC1
UN x.x.x.142 18.71 GB 512 100.0% 1df2908e-5697-4b1a-9110-dcf581511510 RAC1
// First, we trigger deletes on one of the original three nodes.
// Then, we run the below on the bootstrapping node
$ nodetool flush && nodetool compact
$ nodetool compactionhistory
{code}
Compaction History:
{code:java}
id keyspace_name columnfamily_name compacted_at bytes_in bytes_out rows_merged
b63e7e30-d00a-11e9-a7b9-f18b3a65a899 datadump ddt 1567708003731 366768 0 {}
7e2253a0-d00a-11e9-a7b9-f18b3a65a899 system local 1567707909594 10705 10500
{4:1}
{code}
h3. Final State
Once the bootstrap has completed, the cluster looks like this:
{code:java}
$ nodetool status
Datacenter: DC11
Status=Up/Down
|/ State=Normal/Leaving/Joining/Moving
-- Address Load Tokens Owns (effective) Host ID Rack
UN x.x.x.129 14.93 GB 512 76.1% 8418418b-2aa2-4918-a065-8fc25887194f RAC1
UN x.x.x.180 19.54 GB 512 74.5% 5b073424-1f21-4aac-acd5-e0e8e82f7073 RAC1
UN x.x.x.217 19.54 GB 512 75.3% 20e9ecd3-b3a5-4171-901c-eb6e7bc7a429 RAC1
UN x.x.x.142 19.54 GB 512 74.1% 1df2908e-5697-4b1a-9110-dcf581511510 RAC1{code}
We run a flush and major compaction on every node. The original three nodes
drop everything, but the bootstrapped node holds onto nearly 75% of the data.
{code:java}
$ nodetool status
Datacenter: DC11
Status=Up/Down
|/ State=Normal/Leaving/Joining/Moving
-- Address Load Tokens Owns (effective) Host ID Rack
UN x.x.x.129 14.93 GB 512 76.1% 8418418b-2aa2-4918-a065-8fc25887194f RAC1
UN x.x.x.180 143.39 KB 512 74.5% 5b073424-1f21-4aac-acd5-e0e8e82f7073 RAC1
UN x.x.x.217 128.6 KB 512 75.3% 20e9ecd3-b3a5-4171-901c-eb6e7bc7a429 RAC1
UN x.x.x.142 149.97 KB 512 74.1% 1df2908e-5697-4b1a-9110-dcf581511510 RAC1{code}
If we run the count query again, we see the data get read-repaired back into
the other nodes.
{code:java}
$ nodetool status
Datacenter: DC11
Status=Up/Down
|/ State=Normal/Leaving/Joining/Moving
-- Address Load Tokens Owns
[jira] [Updated] (CASSANDRA-15327) Deleted data can re-appear if range movement streaming time exceeds gc_grace_seconds
[
https://issues.apache.org/jira/browse/CASSANDRA-15327?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Leon Zaruvinsky updated CASSANDRA-15327:
Attachment: CASSANDRA-15327-2.1.txt
> Deleted data can re-appear if range movement streaming time exceeds
> gc_grace_seconds
>
>
> Key: CASSANDRA-15327
> URL: https://issues.apache.org/jira/browse/CASSANDRA-15327
> Project: Cassandra
> Issue Type: Bug
> Components: Consistency/Bootstrap and Decommission
>Reporter: Leon Zaruvinsky
>Priority: Normal
> Attachments: CASSANDRA-15327-2.1.txt
>
>
> Hey,
> We've come across a scenario in production (noticed on Cassandra 2.2.14)
> where data that is deleted from Cassandra at consistency {{ALL}} can be
> resurrected. I've added a reproduction in a comment.
> If a {{delete}} is issued during a range movement (i.e. bootstrap,
> decommission, move), and {{gc_grace_seconds}} is surpassed before the stream
> is finished, then the tombstones from the {{delete}} can be purged from the
> recipient node before the data is streamed. Once the move is complete, the
> data now exists on the recipient node without a tombstone.
>
> We noticed this because our bootstrapping time occasionally exceeds our
> configured gc_grace_seconds, so we lose the consistency guarantee. As an
> operator, it would be great to not have to worry about this edge case.
> I've attached a patch that we have tested and successfully used in
> production, and haven't noticed any ill effects.
--
This message was sent by Atlassian Jira
(v8.3.2#803003)
-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Created] (CASSANDRA-15327) Deleted data can re-appear if range movement streaming time exceeds gc_grace_seconds
Leon Zaruvinsky created CASSANDRA-15327:
---
Summary: Deleted data can re-appear if range movement streaming
time exceeds gc_grace_seconds
Key: CASSANDRA-15327
URL: https://issues.apache.org/jira/browse/CASSANDRA-15327
Project: Cassandra
Issue Type: Bug
Components: Consistency/Bootstrap and Decommission
Reporter: Leon Zaruvinsky
Hey,
We've come across a scenario in production (noticed on Cassandra 2.2.14) where
data that is deleted from Cassandra at consistency {{ALL}} can be resurrected.
I've added a reproduction in a comment.
If a {{delete}} is issued during a range movement (i.e. bootstrap,
decommission, move), and {{gc_grace_seconds}} is surpassed before the stream is
finished, then the tombstones from the {{delete}} can be purged from the
recipient node before the data is streamed. Once the move is complete, the data
now exists on the recipient node without a tombstone.
We noticed this because our bootstrapping time occasionally exceeds our
configured gc_grace_seconds, so we lose the consistency guarantee. As an
operator, it would be great to not have to worry about this edge case.
I've attached a patch that we have tested and successfully used in production,
and haven't noticed any ill effects.
--
This message was sent by Atlassian Jira
(v8.3.2#803003)
-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org
