[jira] [Commented] (CASSANDRA-15433) Pending ranges are not recalculated on keyspace creation
[
https://issues.apache.org/jira/browse/CASSANDRA-15433?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17429206#comment-17429206
]
Alex Petrov commented on CASSANDRA-15433:
-
Committed to 3.0 with
[11af037fd99bfb4a942f7d7dd55c177a37d29f63|https://github.com/apache/cassandra/commit/11af037fd99bfb4a942f7d7dd55c177a37d29f63]
and merged up to
[3.11|https://github.com/apache/cassandra/commit/610d5d4eb5205cd4ca2f573237da14db055757c1],
[4.0|https://github.com/apache/cassandra/commit/25f67a75e22fdd9bdabd4b17dd0e11973ce59c2d]
and
[trunk|https://github.com/apache/cassandra/commit/d9ca61404334f3bd94c08cf66ccd15e8c5287f52].
> Pending ranges are not recalculated on keyspace creation
>
>
> Key: CASSANDRA-15433
> URL: https://issues.apache.org/jira/browse/CASSANDRA-15433
> Project: Cassandra
> Issue Type: Bug
> Components: Cluster/Membership
>Reporter: Josh Snyder
>Assignee: Sumanth Pasupuleti
>Priority: Normal
>
> When a node begins bootstrapping, Cassandra recalculates pending tokens for
> each keyspace that exists when the state change is observed (in
> StorageService:handleState*). When new keyspaces are created, we do not
> recalculate pending ranges (around Schema:merge). As a result, writes for new
> keyspaces are not received by nodes in BOOT or BOOT_REPLACE modes. When
> bootstrapping finishes, the node which just bootstrapped will not have data
> for the newly created keyspace.
> Consider a ring with bootstrapped nodes A, B, and C. Node D is pending, and
> when it finishes bootstrapping, C will cede ownership of some ranges to D. A
> quorum write is acknowledged by C and A. B missed the write, and the
> coordinator didn't send it to D at all. When D finishes bootstrapping, the
> quorum B+D will not contain the mutation.
> Steps to reproduce:
> # Join a node in BOOT mode
> # Create a keyspace
> # Send writes to that keyspace
> # On the joining node, observe that {{nodetool cfstats}} records zero writes
> to the new keyspace
> I have observed this directly in Cassandra 3.0, and based on my reading the
> code, I believe it affects up through trunk.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Commented] (CASSANDRA-15433) Pending ranges are not recalculated on keyspace creation
[
https://issues.apache.org/jira/browse/CASSANDRA-15433?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17427960#comment-17427960
]
Sumanth Pasupuleti commented on CASSANDRA-15433:
[~ifesdjeen] curious if you will be able to commit this patch
> Pending ranges are not recalculated on keyspace creation
>
>
> Key: CASSANDRA-15433
> URL: https://issues.apache.org/jira/browse/CASSANDRA-15433
> Project: Cassandra
> Issue Type: Bug
> Components: Cluster/Membership
>Reporter: Josh Snyder
>Assignee: Sumanth Pasupuleti
>Priority: Normal
>
> When a node begins bootstrapping, Cassandra recalculates pending tokens for
> each keyspace that exists when the state change is observed (in
> StorageService:handleState*). When new keyspaces are created, we do not
> recalculate pending ranges (around Schema:merge). As a result, writes for new
> keyspaces are not received by nodes in BOOT or BOOT_REPLACE modes. When
> bootstrapping finishes, the node which just bootstrapped will not have data
> for the newly created keyspace.
> Consider a ring with bootstrapped nodes A, B, and C. Node D is pending, and
> when it finishes bootstrapping, C will cede ownership of some ranges to D. A
> quorum write is acknowledged by C and A. B missed the write, and the
> coordinator didn't send it to D at all. When D finishes bootstrapping, the
> quorum B+D will not contain the mutation.
> Steps to reproduce:
> # Join a node in BOOT mode
> # Create a keyspace
> # Send writes to that keyspace
> # On the joining node, observe that {{nodetool cfstats}} records zero writes
> to the new keyspace
> I have observed this directly in Cassandra 3.0, and based on my reading the
> code, I believe it affects up through trunk.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Commented] (CASSANDRA-15433) Pending ranges are not recalculated on keyspace creation
[
https://issues.apache.org/jira/browse/CASSANDRA-15433?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17403070#comment-17403070
]
Sumanth Pasupuleti commented on CASSANDRA-15433:
[~ifesdjeen] I was finally able to get back to making progress on this ticket.
Please find the updated patches below that include in-jvm tests for this
scenario.
[Trunk
changes|https://github.com/apache/cassandra/compare/trunk...sumanth-pasupuleti:bugfix/trunk_15433?expand=1]
[3.0
changes|https://github.com/apache/cassandra/compare/cassandra-3.0...sumanth-pasupuleti:bugfix/30_15433?expand=1]
> Pending ranges are not recalculated on keyspace creation
>
>
> Key: CASSANDRA-15433
> URL: https://issues.apache.org/jira/browse/CASSANDRA-15433
> Project: Cassandra
> Issue Type: Bug
> Components: Cluster/Membership
>Reporter: Josh Snyder
>Assignee: Sumanth Pasupuleti
>Priority: Normal
>
> When a node begins bootstrapping, Cassandra recalculates pending tokens for
> each keyspace that exists when the state change is observed (in
> StorageService:handleState*). When new keyspaces are created, we do not
> recalculate pending ranges (around Schema:merge). As a result, writes for new
> keyspaces are not received by nodes in BOOT or BOOT_REPLACE modes. When
> bootstrapping finishes, the node which just bootstrapped will not have data
> for the newly created keyspace.
> Consider a ring with bootstrapped nodes A, B, and C. Node D is pending, and
> when it finishes bootstrapping, C will cede ownership of some ranges to D. A
> quorum write is acknowledged by C and A. B missed the write, and the
> coordinator didn't send it to D at all. When D finishes bootstrapping, the
> quorum B+D will not contain the mutation.
> Steps to reproduce:
> # Join a node in BOOT mode
> # Create a keyspace
> # Send writes to that keyspace
> # On the joining node, observe that {{nodetool cfstats}} records zero writes
> to the new keyspace
> I have observed this directly in Cassandra 3.0, and based on my reading the
> code, I believe it affects up through trunk.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Commented] (CASSANDRA-15433) Pending ranges are not recalculated on keyspace creation
[
https://issues.apache.org/jira/browse/CASSANDRA-15433?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17038079#comment-17038079
]
Sumanth Pasupuleti commented on CASSANDRA-15433:
+1 on invoking pendingrange calculation only for newly created keyspaces. I
have updated 3.0 and trunk changes accordingly.
[3.0
changes|https://github.com/sumanth-pasupuleti/cassandra/commit/d2841ed2075db9e7241d428a054cf1d7d2936f9e]
[Trunk changes
|https://github.com/sumanth-pasupuleti/cassandra/commit/dac07262b23e96607fb12a869f84b55a4081ef36]
I am still working on in-jvm test. As I was mentioning offline, trying to
figure out how to make a node to be in "bootstrapping mode" in order to then
trigger a keyspace creation while the node is in bootstrapping mode.
> Pending ranges are not recalculated on keyspace creation
>
>
> Key: CASSANDRA-15433
> URL: https://issues.apache.org/jira/browse/CASSANDRA-15433
> Project: Cassandra
> Issue Type: Bug
> Components: Cluster/Membership
>Reporter: Josh Snyder
>Assignee: Sumanth Pasupuleti
>Priority: Normal
>
> When a node begins bootstrapping, Cassandra recalculates pending tokens for
> each keyspace that exists when the state change is observed (in
> StorageService:handleState*). When new keyspaces are created, we do not
> recalculate pending ranges (around Schema:merge). As a result, writes for new
> keyspaces are not received by nodes in BOOT or BOOT_REPLACE modes. When
> bootstrapping finishes, the node which just bootstrapped will not have data
> for the newly created keyspace.
> Consider a ring with bootstrapped nodes A, B, and C. Node D is pending, and
> when it finishes bootstrapping, C will cede ownership of some ranges to D. A
> quorum write is acknowledged by C and A. B missed the write, and the
> coordinator didn't send it to D at all. When D finishes bootstrapping, the
> quorum B+D will not contain the mutation.
> Steps to reproduce:
> # Join a node in BOOT mode
> # Create a keyspace
> # Send writes to that keyspace
> # On the joining node, observe that {{nodetool cfstats}} records zero writes
> to the new keyspace
> I have observed this directly in Cassandra 3.0, and based on my reading the
> code, I believe it affects up through trunk.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Commented] (CASSANDRA-15433) Pending ranges are not recalculated on keyspace creation
[
https://issues.apache.org/jira/browse/CASSANDRA-15433?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17015172#comment-17015172
]
Alex Petrov commented on CASSANDRA-15433:
-
Thank you for the patch.
Following up an offline discussion, I've submitted a patch that you can use to
create a test for this patch (see dependent jira) that could also help to
simplify further development of features related to bootstrap and ring
movements.
While what you are suggesting seems to mitigate the problem I'm not fully
certain that this is what we need to do. First, pending range tasks will be
triggered for _all_ keyspaces, which is definitely not what we want to do. From
looking at the code, it seems like for newly created keyspaces, we do not
calculate pending ranges at all, so I'd recompute them for one of the keyspaces
(namely, for freshly created one).
Moreover, keyspaces diff is more than just created keyspaces. It also contains
created and altered ones, so triggering recompute without a regard to what
exactly happened to which keyspace is also not what we'd like.
To summarise, I'd say we need to add a test for bootstrap + keyspace creation;
technically it'll also cover moving and leaving nodes since we also use pending
ranges there. And we need to make sure we recompute pending ranges only for the
keyspaces that were created.
> Pending ranges are not recalculated on keyspace creation
>
>
> Key: CASSANDRA-15433
> URL: https://issues.apache.org/jira/browse/CASSANDRA-15433
> Project: Cassandra
> Issue Type: Bug
> Components: Cluster/Membership
>Reporter: Josh Snyder
>Assignee: Sumanth Pasupuleti
>Priority: Normal
>
> When a node begins bootstrapping, Cassandra recalculates pending tokens for
> each keyspace that exists when the state change is observed (in
> StorageService:handleState*). When new keyspaces are created, we do not
> recalculate pending ranges (around Schema:merge). As a result, writes for new
> keyspaces are not received by nodes in BOOT or BOOT_REPLACE modes. When
> bootstrapping finishes, the node which just bootstrapped will not have data
> for the newly created keyspace.
> Consider a ring with bootstrapped nodes A, B, and C. Node D is pending, and
> when it finishes bootstrapping, C will cede ownership of some ranges to D. A
> quorum write is acknowledged by C and A. B missed the write, and the
> coordinator didn't send it to D at all. When D finishes bootstrapping, the
> quorum B+D will not contain the mutation.
> Steps to reproduce:
> # Join a node in BOOT mode
> # Create a keyspace
> # Send writes to that keyspace
> # On the joining node, observe that {{nodetool cfstats}} records zero writes
> to the new keyspace
> I have observed this directly in Cassandra 3.0, and based on my reading the
> code, I believe it affects up through trunk.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Commented] (CASSANDRA-15433) Pending ranges are not recalculated on keyspace creation
[
https://issues.apache.org/jira/browse/CASSANDRA-15433?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17002137#comment-17002137
]
Sumanth Pasupuleti commented on CASSANDRA-15433:
A trivial approach would be to recalculate pending ranges whenever keyspaces
change.
https://github.com/apache/cassandra/compare/cassandra-3.0...sumanth-pasupuleti:30_15433?expand=1
Added a blocking wait to ensure keyspace change isn't complete unless pending
changes are recalculated to avoid any potential miss of mutations on the
bootstrapping nodes.
> Pending ranges are not recalculated on keyspace creation
>
>
> Key: CASSANDRA-15433
> URL: https://issues.apache.org/jira/browse/CASSANDRA-15433
> Project: Cassandra
> Issue Type: Bug
> Components: Cluster/Membership
>Reporter: Josh Snyder
>Assignee: Sumanth Pasupuleti
>Priority: Normal
>
> When a node begins bootstrapping, Cassandra recalculates pending tokens for
> each keyspace that exists when the state change is observed (in
> StorageService:handleState*). When new keyspaces are created, we do not
> recalculate pending ranges (around Schema:merge). As a result, writes for new
> keyspaces are not received by nodes in BOOT or BOOT_REPLACE modes. When
> bootstrapping finishes, the node which just bootstrapped will not have data
> for the newly created keyspace.
> Consider a ring with bootstrapped nodes A, B, and C. Node D is pending, and
> when it finishes bootstrapping, C will cede ownership of some ranges to D. A
> quorum write is acknowledged by C and A. B missed the write, and the
> coordinator didn't send it to D at all. When D finishes bootstrapping, the
> quorum B+D will not contain the mutation.
> Steps to reproduce:
> # Join a node in BOOT mode
> # Create a keyspace
> # Send writes to that keyspace
> # On the joining node, observe that {{nodetool cfstats}} records zero writes
> to the new keyspace
> I have observed this directly in Cassandra 3.0, and based on my reading the
> code, I believe it affects up through trunk.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org
