[jira] [Updated] (CASSANDRA-15439) Token metadata for bootstrapping nodes is lost under temporary failures
[
https://issues.apache.org/jira/browse/CASSANDRA-15439?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Brandon Williams updated CASSANDRA-15439:
-
Fix Version/s: 4.0.14
4.1.6
5.0-beta2
5.0
(was: 4.0.x)
(was: 4.1.x)
(was: 5.0.x)
Since Version: NA
Source Control Link:
https://github.com/apache/cassandra/commit/057d082e00f7d10b8e9b127cfabd9b8cd228da3d
Resolution: Fixed
Status: Resolved (was: Ready to Commit)
Committed, thanks all!
> Token metadata for bootstrapping nodes is lost under temporary failures
> ---
>
> Key: CASSANDRA-15439
> URL: https://issues.apache.org/jira/browse/CASSANDRA-15439
> Project: Cassandra
> Issue Type: Bug
> Components: Cluster/Membership
>Reporter: Josh Snyder
>Assignee: Raymond Huffman
>Priority: Normal
> Fix For: 4.0.14, 4.1.6, 5.0-beta2, 5.0
>
> Time Spent: 1h
> Remaining Estimate: 0h
>
> In CASSANDRA-8838, [~pauloricardomg] asked "hints will not be stored to the
> bootstrapping node after RING_DELAY, since it will evicted from the TMD
> pending ranges. Should we create a ticket to address this?"
> CASSANDRA-15264 relates to the most likely cause of such situations, where
> the Cassandra daemon on the bootstrapping node completely crashes. Based on
> testing with {{kill -STOP}} on a bootstrapping Cassandra JVM, I believe it
> also is possible to remove token metadata (and thus pending ranges, and thus
> hints) for a bootstrapping node, simply by affecting its status in the
> failure detector.
> A node in the cluster sees the bootstrapping node this way:
> {noformat}
> INFO [GossipStage:1] 2019-11-27 20:41:41,101 Gossiper.java: - Node
> /PUBLIC-IP is now part of the cluster
> INFO [GossipStage:1] 2019-11-27 20:41:41,199 Gossiper.java:1073 -
> InetAddress /PUBLIC-IP is now UP
> INFO [HANDSHAKE-/PRIVATE-IP] 2019-11-27 20:41:41,412
> OutboundTcpConnection.java:565 - Handshaking version with /PRIVATE-IP
> INFO [STREAM-INIT-/PRIVATE-IP:21233] 2019-11-27 20:42:10,019
> StreamResultFuture.java:112 - [Stream #6219a950-1156-11ea-b45d-4d30364576c4
> ID#0] Creating new streaming plan for Bootstrap
> INFO [STREAM-INIT-/PRIVATE-IP:21233] 2019-11-27 20:42:10,020
> StreamResultFuture.java:119 - [Stream #6219a950-1156-11ea-b45d-4d30364576c4,
> ID#0] Received streaming plan for Bootstrap
> INFO [STREAM-INIT-/PRIVATE-IP:56003] 2019-11-27 20:42:10,112
> StreamResultFuture.java:119 - [Stream #6219a950-1156-11ea-b45d-4d30364576c4,
> ID#0] Received streaming plan for Bootstrap
> INFO [STREAM-IN-/PUBLIC-IP] 2019-11-27 20:42:10,179
> StreamResultFuture.java:169 - [Stream #6219a950-1156-11ea-b45d-4d30364576c4
> ID#0] Prepare completed. Receiving 0 files(0 bytes), sending 833
> files(139744616815 bytes)
> INFO [GossipStage:1] 2019-11-27 20:54:47,547 Gossiper.java:1089 -
> InetAddress /PUBLIC-IP is now DOWN
> INFO [GossipTasks:1] 2019-11-27 20:54:57,551 Gossiper.java:849 - FatClient
> /PUBLIC-IP has been silent for 3ms, removing from gossip
> {noformat}
> Since the bootstrapping node has no tokens, it is treated like a fat client,
> and it is removed from the ring. For correctness purposes, I believe we must
> keep storing hints for the downed bootstrapping node until it is either
> assassinated or until a replacement attempts to bootstrap for the same token.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Updated] (CASSANDRA-15439) Token metadata for bootstrapping nodes is lost under temporary failures
[
https://issues.apache.org/jira/browse/CASSANDRA-15439?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Brandon Williams updated CASSANDRA-15439:
-
Status: Ready to Commit (was: Review In Progress)
> Token metadata for bootstrapping nodes is lost under temporary failures
> ---
>
> Key: CASSANDRA-15439
> URL: https://issues.apache.org/jira/browse/CASSANDRA-15439
> Project: Cassandra
> Issue Type: Bug
> Components: Cluster/Membership
>Reporter: Josh Snyder
>Assignee: Raymond Huffman
>Priority: Normal
> Fix For: 4.0.x, 4.1.x, 5.0.x
>
> Time Spent: 1h
> Remaining Estimate: 0h
>
> In CASSANDRA-8838, [~pauloricardomg] asked "hints will not be stored to the
> bootstrapping node after RING_DELAY, since it will evicted from the TMD
> pending ranges. Should we create a ticket to address this?"
> CASSANDRA-15264 relates to the most likely cause of such situations, where
> the Cassandra daemon on the bootstrapping node completely crashes. Based on
> testing with {{kill -STOP}} on a bootstrapping Cassandra JVM, I believe it
> also is possible to remove token metadata (and thus pending ranges, and thus
> hints) for a bootstrapping node, simply by affecting its status in the
> failure detector.
> A node in the cluster sees the bootstrapping node this way:
> {noformat}
> INFO [GossipStage:1] 2019-11-27 20:41:41,101 Gossiper.java: - Node
> /PUBLIC-IP is now part of the cluster
> INFO [GossipStage:1] 2019-11-27 20:41:41,199 Gossiper.java:1073 -
> InetAddress /PUBLIC-IP is now UP
> INFO [HANDSHAKE-/PRIVATE-IP] 2019-11-27 20:41:41,412
> OutboundTcpConnection.java:565 - Handshaking version with /PRIVATE-IP
> INFO [STREAM-INIT-/PRIVATE-IP:21233] 2019-11-27 20:42:10,019
> StreamResultFuture.java:112 - [Stream #6219a950-1156-11ea-b45d-4d30364576c4
> ID#0] Creating new streaming plan for Bootstrap
> INFO [STREAM-INIT-/PRIVATE-IP:21233] 2019-11-27 20:42:10,020
> StreamResultFuture.java:119 - [Stream #6219a950-1156-11ea-b45d-4d30364576c4,
> ID#0] Received streaming plan for Bootstrap
> INFO [STREAM-INIT-/PRIVATE-IP:56003] 2019-11-27 20:42:10,112
> StreamResultFuture.java:119 - [Stream #6219a950-1156-11ea-b45d-4d30364576c4,
> ID#0] Received streaming plan for Bootstrap
> INFO [STREAM-IN-/PUBLIC-IP] 2019-11-27 20:42:10,179
> StreamResultFuture.java:169 - [Stream #6219a950-1156-11ea-b45d-4d30364576c4
> ID#0] Prepare completed. Receiving 0 files(0 bytes), sending 833
> files(139744616815 bytes)
> INFO [GossipStage:1] 2019-11-27 20:54:47,547 Gossiper.java:1089 -
> InetAddress /PUBLIC-IP is now DOWN
> INFO [GossipTasks:1] 2019-11-27 20:54:57,551 Gossiper.java:849 - FatClient
> /PUBLIC-IP has been silent for 3ms, removing from gossip
> {noformat}
> Since the bootstrapping node has no tokens, it is treated like a fat client,
> and it is removed from the ring. For correctness purposes, I believe we must
> keep storing hints for the downed bootstrapping node until it is either
> assassinated or until a replacement attempts to bootstrap for the same token.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Updated] (CASSANDRA-15439) Token metadata for bootstrapping nodes is lost under temporary failures
[
https://issues.apache.org/jira/browse/CASSANDRA-15439?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Brandon Williams updated CASSANDRA-15439:
-
Reviewers: Brandon Williams, David Capwell (was: Brandon Williams)
Status: Review In Progress (was: Needs Committer)
> Token metadata for bootstrapping nodes is lost under temporary failures
> ---
>
> Key: CASSANDRA-15439
> URL: https://issues.apache.org/jira/browse/CASSANDRA-15439
> Project: Cassandra
> Issue Type: Bug
> Components: Cluster/Membership
>Reporter: Josh Snyder
>Assignee: Raymond Huffman
>Priority: Normal
> Fix For: 4.0.x, 4.1.x, 5.0.x
>
> Time Spent: 1h
> Remaining Estimate: 0h
>
> In CASSANDRA-8838, [~pauloricardomg] asked "hints will not be stored to the
> bootstrapping node after RING_DELAY, since it will evicted from the TMD
> pending ranges. Should we create a ticket to address this?"
> CASSANDRA-15264 relates to the most likely cause of such situations, where
> the Cassandra daemon on the bootstrapping node completely crashes. Based on
> testing with {{kill -STOP}} on a bootstrapping Cassandra JVM, I believe it
> also is possible to remove token metadata (and thus pending ranges, and thus
> hints) for a bootstrapping node, simply by affecting its status in the
> failure detector.
> A node in the cluster sees the bootstrapping node this way:
> {noformat}
> INFO [GossipStage:1] 2019-11-27 20:41:41,101 Gossiper.java: - Node
> /PUBLIC-IP is now part of the cluster
> INFO [GossipStage:1] 2019-11-27 20:41:41,199 Gossiper.java:1073 -
> InetAddress /PUBLIC-IP is now UP
> INFO [HANDSHAKE-/PRIVATE-IP] 2019-11-27 20:41:41,412
> OutboundTcpConnection.java:565 - Handshaking version with /PRIVATE-IP
> INFO [STREAM-INIT-/PRIVATE-IP:21233] 2019-11-27 20:42:10,019
> StreamResultFuture.java:112 - [Stream #6219a950-1156-11ea-b45d-4d30364576c4
> ID#0] Creating new streaming plan for Bootstrap
> INFO [STREAM-INIT-/PRIVATE-IP:21233] 2019-11-27 20:42:10,020
> StreamResultFuture.java:119 - [Stream #6219a950-1156-11ea-b45d-4d30364576c4,
> ID#0] Received streaming plan for Bootstrap
> INFO [STREAM-INIT-/PRIVATE-IP:56003] 2019-11-27 20:42:10,112
> StreamResultFuture.java:119 - [Stream #6219a950-1156-11ea-b45d-4d30364576c4,
> ID#0] Received streaming plan for Bootstrap
> INFO [STREAM-IN-/PUBLIC-IP] 2019-11-27 20:42:10,179
> StreamResultFuture.java:169 - [Stream #6219a950-1156-11ea-b45d-4d30364576c4
> ID#0] Prepare completed. Receiving 0 files(0 bytes), sending 833
> files(139744616815 bytes)
> INFO [GossipStage:1] 2019-11-27 20:54:47,547 Gossiper.java:1089 -
> InetAddress /PUBLIC-IP is now DOWN
> INFO [GossipTasks:1] 2019-11-27 20:54:57,551 Gossiper.java:849 - FatClient
> /PUBLIC-IP has been silent for 3ms, removing from gossip
> {noformat}
> Since the bootstrapping node has no tokens, it is treated like a fat client,
> and it is removed from the ring. For correctness purposes, I believe we must
> keep storing hints for the downed bootstrapping node until it is either
> assassinated or until a replacement attempts to bootstrap for the same token.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Updated] (CASSANDRA-15439) Token metadata for bootstrapping nodes is lost under temporary failures
[
https://issues.apache.org/jira/browse/CASSANDRA-15439?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Brandon Williams updated CASSANDRA-15439:
-
Status: Needs Committer (was: Patch Available)
> Token metadata for bootstrapping nodes is lost under temporary failures
> ---
>
> Key: CASSANDRA-15439
> URL: https://issues.apache.org/jira/browse/CASSANDRA-15439
> Project: Cassandra
> Issue Type: Bug
> Components: Cluster/Membership
>Reporter: Josh Snyder
>Assignee: Raymond Huffman
>Priority: Normal
> Fix For: 4.0.x, 4.1.x, 5.0.x
>
> Time Spent: 20m
> Remaining Estimate: 0h
>
> In CASSANDRA-8838, [~pauloricardomg] asked "hints will not be stored to the
> bootstrapping node after RING_DELAY, since it will evicted from the TMD
> pending ranges. Should we create a ticket to address this?"
> CASSANDRA-15264 relates to the most likely cause of such situations, where
> the Cassandra daemon on the bootstrapping node completely crashes. Based on
> testing with {{kill -STOP}} on a bootstrapping Cassandra JVM, I believe it
> also is possible to remove token metadata (and thus pending ranges, and thus
> hints) for a bootstrapping node, simply by affecting its status in the
> failure detector.
> A node in the cluster sees the bootstrapping node this way:
> {noformat}
> INFO [GossipStage:1] 2019-11-27 20:41:41,101 Gossiper.java: - Node
> /PUBLIC-IP is now part of the cluster
> INFO [GossipStage:1] 2019-11-27 20:41:41,199 Gossiper.java:1073 -
> InetAddress /PUBLIC-IP is now UP
> INFO [HANDSHAKE-/PRIVATE-IP] 2019-11-27 20:41:41,412
> OutboundTcpConnection.java:565 - Handshaking version with /PRIVATE-IP
> INFO [STREAM-INIT-/PRIVATE-IP:21233] 2019-11-27 20:42:10,019
> StreamResultFuture.java:112 - [Stream #6219a950-1156-11ea-b45d-4d30364576c4
> ID#0] Creating new streaming plan for Bootstrap
> INFO [STREAM-INIT-/PRIVATE-IP:21233] 2019-11-27 20:42:10,020
> StreamResultFuture.java:119 - [Stream #6219a950-1156-11ea-b45d-4d30364576c4,
> ID#0] Received streaming plan for Bootstrap
> INFO [STREAM-INIT-/PRIVATE-IP:56003] 2019-11-27 20:42:10,112
> StreamResultFuture.java:119 - [Stream #6219a950-1156-11ea-b45d-4d30364576c4,
> ID#0] Received streaming plan for Bootstrap
> INFO [STREAM-IN-/PUBLIC-IP] 2019-11-27 20:42:10,179
> StreamResultFuture.java:169 - [Stream #6219a950-1156-11ea-b45d-4d30364576c4
> ID#0] Prepare completed. Receiving 0 files(0 bytes), sending 833
> files(139744616815 bytes)
> INFO [GossipStage:1] 2019-11-27 20:54:47,547 Gossiper.java:1089 -
> InetAddress /PUBLIC-IP is now DOWN
> INFO [GossipTasks:1] 2019-11-27 20:54:57,551 Gossiper.java:849 - FatClient
> /PUBLIC-IP has been silent for 3ms, removing from gossip
> {noformat}
> Since the bootstrapping node has no tokens, it is treated like a fat client,
> and it is removed from the ring. For correctness purposes, I believe we must
> keep storing hints for the downed bootstrapping node until it is either
> assassinated or until a replacement attempts to bootstrap for the same token.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Updated] (CASSANDRA-15439) Token metadata for bootstrapping nodes is lost under temporary failures
[
https://issues.apache.org/jira/browse/CASSANDRA-15439?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Brandon Williams updated CASSANDRA-15439:
-
Test and Documentation Plan: run CI
Status: Patch Available (was: Open)
> Token metadata for bootstrapping nodes is lost under temporary failures
> ---
>
> Key: CASSANDRA-15439
> URL: https://issues.apache.org/jira/browse/CASSANDRA-15439
> Project: Cassandra
> Issue Type: Bug
> Components: Cluster/Membership
>Reporter: Josh Snyder
>Assignee: Raymond Huffman
>Priority: Normal
> Fix For: 4.0.x, 4.1.x, 5.0.x
>
> Time Spent: 20m
> Remaining Estimate: 0h
>
> In CASSANDRA-8838, [~pauloricardomg] asked "hints will not be stored to the
> bootstrapping node after RING_DELAY, since it will evicted from the TMD
> pending ranges. Should we create a ticket to address this?"
> CASSANDRA-15264 relates to the most likely cause of such situations, where
> the Cassandra daemon on the bootstrapping node completely crashes. Based on
> testing with {{kill -STOP}} on a bootstrapping Cassandra JVM, I believe it
> also is possible to remove token metadata (and thus pending ranges, and thus
> hints) for a bootstrapping node, simply by affecting its status in the
> failure detector.
> A node in the cluster sees the bootstrapping node this way:
> {noformat}
> INFO [GossipStage:1] 2019-11-27 20:41:41,101 Gossiper.java: - Node
> /PUBLIC-IP is now part of the cluster
> INFO [GossipStage:1] 2019-11-27 20:41:41,199 Gossiper.java:1073 -
> InetAddress /PUBLIC-IP is now UP
> INFO [HANDSHAKE-/PRIVATE-IP] 2019-11-27 20:41:41,412
> OutboundTcpConnection.java:565 - Handshaking version with /PRIVATE-IP
> INFO [STREAM-INIT-/PRIVATE-IP:21233] 2019-11-27 20:42:10,019
> StreamResultFuture.java:112 - [Stream #6219a950-1156-11ea-b45d-4d30364576c4
> ID#0] Creating new streaming plan for Bootstrap
> INFO [STREAM-INIT-/PRIVATE-IP:21233] 2019-11-27 20:42:10,020
> StreamResultFuture.java:119 - [Stream #6219a950-1156-11ea-b45d-4d30364576c4,
> ID#0] Received streaming plan for Bootstrap
> INFO [STREAM-INIT-/PRIVATE-IP:56003] 2019-11-27 20:42:10,112
> StreamResultFuture.java:119 - [Stream #6219a950-1156-11ea-b45d-4d30364576c4,
> ID#0] Received streaming plan for Bootstrap
> INFO [STREAM-IN-/PUBLIC-IP] 2019-11-27 20:42:10,179
> StreamResultFuture.java:169 - [Stream #6219a950-1156-11ea-b45d-4d30364576c4
> ID#0] Prepare completed. Receiving 0 files(0 bytes), sending 833
> files(139744616815 bytes)
> INFO [GossipStage:1] 2019-11-27 20:54:47,547 Gossiper.java:1089 -
> InetAddress /PUBLIC-IP is now DOWN
> INFO [GossipTasks:1] 2019-11-27 20:54:57,551 Gossiper.java:849 - FatClient
> /PUBLIC-IP has been silent for 3ms, removing from gossip
> {noformat}
> Since the bootstrapping node has no tokens, it is treated like a fat client,
> and it is removed from the ring. For correctness purposes, I believe we must
> keep storing hints for the downed bootstrapping node until it is either
> assassinated or until a replacement attempts to bootstrap for the same token.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Updated] (CASSANDRA-15439) Token metadata for bootstrapping nodes is lost under temporary failures
[
https://issues.apache.org/jira/browse/CASSANDRA-15439?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Brandon Williams updated CASSANDRA-15439:
-
Reviewers: Brandon Williams
> Token metadata for bootstrapping nodes is lost under temporary failures
> ---
>
> Key: CASSANDRA-15439
> URL: https://issues.apache.org/jira/browse/CASSANDRA-15439
> Project: Cassandra
> Issue Type: Bug
> Components: Cluster/Membership
>Reporter: Josh Snyder
>Assignee: Raymond Huffman
>Priority: Normal
> Fix For: 4.0.x, 4.1.x, 5.0.x
>
> Time Spent: 20m
> Remaining Estimate: 0h
>
> In CASSANDRA-8838, [~pauloricardomg] asked "hints will not be stored to the
> bootstrapping node after RING_DELAY, since it will evicted from the TMD
> pending ranges. Should we create a ticket to address this?"
> CASSANDRA-15264 relates to the most likely cause of such situations, where
> the Cassandra daemon on the bootstrapping node completely crashes. Based on
> testing with {{kill -STOP}} on a bootstrapping Cassandra JVM, I believe it
> also is possible to remove token metadata (and thus pending ranges, and thus
> hints) for a bootstrapping node, simply by affecting its status in the
> failure detector.
> A node in the cluster sees the bootstrapping node this way:
> {noformat}
> INFO [GossipStage:1] 2019-11-27 20:41:41,101 Gossiper.java: - Node
> /PUBLIC-IP is now part of the cluster
> INFO [GossipStage:1] 2019-11-27 20:41:41,199 Gossiper.java:1073 -
> InetAddress /PUBLIC-IP is now UP
> INFO [HANDSHAKE-/PRIVATE-IP] 2019-11-27 20:41:41,412
> OutboundTcpConnection.java:565 - Handshaking version with /PRIVATE-IP
> INFO [STREAM-INIT-/PRIVATE-IP:21233] 2019-11-27 20:42:10,019
> StreamResultFuture.java:112 - [Stream #6219a950-1156-11ea-b45d-4d30364576c4
> ID#0] Creating new streaming plan for Bootstrap
> INFO [STREAM-INIT-/PRIVATE-IP:21233] 2019-11-27 20:42:10,020
> StreamResultFuture.java:119 - [Stream #6219a950-1156-11ea-b45d-4d30364576c4,
> ID#0] Received streaming plan for Bootstrap
> INFO [STREAM-INIT-/PRIVATE-IP:56003] 2019-11-27 20:42:10,112
> StreamResultFuture.java:119 - [Stream #6219a950-1156-11ea-b45d-4d30364576c4,
> ID#0] Received streaming plan for Bootstrap
> INFO [STREAM-IN-/PUBLIC-IP] 2019-11-27 20:42:10,179
> StreamResultFuture.java:169 - [Stream #6219a950-1156-11ea-b45d-4d30364576c4
> ID#0] Prepare completed. Receiving 0 files(0 bytes), sending 833
> files(139744616815 bytes)
> INFO [GossipStage:1] 2019-11-27 20:54:47,547 Gossiper.java:1089 -
> InetAddress /PUBLIC-IP is now DOWN
> INFO [GossipTasks:1] 2019-11-27 20:54:57,551 Gossiper.java:849 - FatClient
> /PUBLIC-IP has been silent for 3ms, removing from gossip
> {noformat}
> Since the bootstrapping node has no tokens, it is treated like a fat client,
> and it is removed from the ring. For correctness purposes, I believe we must
> keep storing hints for the downed bootstrapping node until it is either
> assassinated or until a replacement attempts to bootstrap for the same token.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Updated] (CASSANDRA-15439) Token metadata for bootstrapping nodes is lost under temporary failures
[
https://issues.apache.org/jira/browse/CASSANDRA-15439?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Brandon Williams updated CASSANDRA-15439:
-
Fix Version/s: (was: 3.0.x)
(was: 3.11.x)
(was: 5.x)
> Token metadata for bootstrapping nodes is lost under temporary failures
> ---
>
> Key: CASSANDRA-15439
> URL: https://issues.apache.org/jira/browse/CASSANDRA-15439
> Project: Cassandra
> Issue Type: Bug
> Components: Cluster/Membership
>Reporter: Josh Snyder
>Priority: Normal
> Fix For: 4.0.x, 4.1.x, 5.0.x
>
> Time Spent: 20m
> Remaining Estimate: 0h
>
> In CASSANDRA-8838, [~pauloricardomg] asked "hints will not be stored to the
> bootstrapping node after RING_DELAY, since it will evicted from the TMD
> pending ranges. Should we create a ticket to address this?"
> CASSANDRA-15264 relates to the most likely cause of such situations, where
> the Cassandra daemon on the bootstrapping node completely crashes. Based on
> testing with {{kill -STOP}} on a bootstrapping Cassandra JVM, I believe it
> also is possible to remove token metadata (and thus pending ranges, and thus
> hints) for a bootstrapping node, simply by affecting its status in the
> failure detector.
> A node in the cluster sees the bootstrapping node this way:
> {noformat}
> INFO [GossipStage:1] 2019-11-27 20:41:41,101 Gossiper.java: - Node
> /PUBLIC-IP is now part of the cluster
> INFO [GossipStage:1] 2019-11-27 20:41:41,199 Gossiper.java:1073 -
> InetAddress /PUBLIC-IP is now UP
> INFO [HANDSHAKE-/PRIVATE-IP] 2019-11-27 20:41:41,412
> OutboundTcpConnection.java:565 - Handshaking version with /PRIVATE-IP
> INFO [STREAM-INIT-/PRIVATE-IP:21233] 2019-11-27 20:42:10,019
> StreamResultFuture.java:112 - [Stream #6219a950-1156-11ea-b45d-4d30364576c4
> ID#0] Creating new streaming plan for Bootstrap
> INFO [STREAM-INIT-/PRIVATE-IP:21233] 2019-11-27 20:42:10,020
> StreamResultFuture.java:119 - [Stream #6219a950-1156-11ea-b45d-4d30364576c4,
> ID#0] Received streaming plan for Bootstrap
> INFO [STREAM-INIT-/PRIVATE-IP:56003] 2019-11-27 20:42:10,112
> StreamResultFuture.java:119 - [Stream #6219a950-1156-11ea-b45d-4d30364576c4,
> ID#0] Received streaming plan for Bootstrap
> INFO [STREAM-IN-/PUBLIC-IP] 2019-11-27 20:42:10,179
> StreamResultFuture.java:169 - [Stream #6219a950-1156-11ea-b45d-4d30364576c4
> ID#0] Prepare completed. Receiving 0 files(0 bytes), sending 833
> files(139744616815 bytes)
> INFO [GossipStage:1] 2019-11-27 20:54:47,547 Gossiper.java:1089 -
> InetAddress /PUBLIC-IP is now DOWN
> INFO [GossipTasks:1] 2019-11-27 20:54:57,551 Gossiper.java:849 - FatClient
> /PUBLIC-IP has been silent for 3ms, removing from gossip
> {noformat}
> Since the bootstrapping node has no tokens, it is treated like a fat client,
> and it is removed from the ring. For correctness purposes, I believe we must
> keep storing hints for the downed bootstrapping node until it is either
> assassinated or until a replacement attempts to bootstrap for the same token.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Updated] (CASSANDRA-15439) Token metadata for bootstrapping nodes is lost under temporary failures
[
https://issues.apache.org/jira/browse/CASSANDRA-15439?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Brandon Williams updated CASSANDRA-15439:
-
Bug Category: Parent values: Correctness(12982)Level 1 values: Recoverable
Corruption / Loss(12986)
Complexity: Normal
Component/s: Cluster/Membership
Discovered By: User Report
Fix Version/s: 3.0.x
3.11.x
4.0.x
4.1.x
5.0.x
5.x
Severity: Normal
Status: Open (was: Triage Needed)
> Token metadata for bootstrapping nodes is lost under temporary failures
> ---
>
> Key: CASSANDRA-15439
> URL: https://issues.apache.org/jira/browse/CASSANDRA-15439
> Project: Cassandra
> Issue Type: Bug
> Components: Cluster/Membership
>Reporter: Josh Snyder
>Priority: Normal
> Fix For: 3.0.x, 3.11.x, 4.0.x, 4.1.x, 5.0.x, 5.x
>
>
> In CASSANDRA-8838, [~pauloricardomg] asked "hints will not be stored to the
> bootstrapping node after RING_DELAY, since it will evicted from the TMD
> pending ranges. Should we create a ticket to address this?"
> CASSANDRA-15264 relates to the most likely cause of such situations, where
> the Cassandra daemon on the bootstrapping node completely crashes. Based on
> testing with {{kill -STOP}} on a bootstrapping Cassandra JVM, I believe it
> also is possible to remove token metadata (and thus pending ranges, and thus
> hints) for a bootstrapping node, simply by affecting its status in the
> failure detector.
> A node in the cluster sees the bootstrapping node this way:
> {noformat}
> INFO [GossipStage:1] 2019-11-27 20:41:41,101 Gossiper.java: - Node
> /PUBLIC-IP is now part of the cluster
> INFO [GossipStage:1] 2019-11-27 20:41:41,199 Gossiper.java:1073 -
> InetAddress /PUBLIC-IP is now UP
> INFO [HANDSHAKE-/PRIVATE-IP] 2019-11-27 20:41:41,412
> OutboundTcpConnection.java:565 - Handshaking version with /PRIVATE-IP
> INFO [STREAM-INIT-/PRIVATE-IP:21233] 2019-11-27 20:42:10,019
> StreamResultFuture.java:112 - [Stream #6219a950-1156-11ea-b45d-4d30364576c4
> ID#0] Creating new streaming plan for Bootstrap
> INFO [STREAM-INIT-/PRIVATE-IP:21233] 2019-11-27 20:42:10,020
> StreamResultFuture.java:119 - [Stream #6219a950-1156-11ea-b45d-4d30364576c4,
> ID#0] Received streaming plan for Bootstrap
> INFO [STREAM-INIT-/PRIVATE-IP:56003] 2019-11-27 20:42:10,112
> StreamResultFuture.java:119 - [Stream #6219a950-1156-11ea-b45d-4d30364576c4,
> ID#0] Received streaming plan for Bootstrap
> INFO [STREAM-IN-/PUBLIC-IP] 2019-11-27 20:42:10,179
> StreamResultFuture.java:169 - [Stream #6219a950-1156-11ea-b45d-4d30364576c4
> ID#0] Prepare completed. Receiving 0 files(0 bytes), sending 833
> files(139744616815 bytes)
> INFO [GossipStage:1] 2019-11-27 20:54:47,547 Gossiper.java:1089 -
> InetAddress /PUBLIC-IP is now DOWN
> INFO [GossipTasks:1] 2019-11-27 20:54:57,551 Gossiper.java:849 - FatClient
> /PUBLIC-IP has been silent for 3ms, removing from gossip
> {noformat}
> Since the bootstrapping node has no tokens, it is treated like a fat client,
> and it is removed from the ring. For correctness purposes, I believe we must
> keep storing hints for the downed bootstrapping node until it is either
> assassinated or until a replacement attempts to bootstrap for the same token.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Updated] (CASSANDRA-15439) Token metadata for bootstrapping nodes is lost under temporary failures
[
https://issues.apache.org/jira/browse/CASSANDRA-15439?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Josh Snyder updated CASSANDRA-15439:
Summary: Token metadata for bootstrapping nodes is lost under temporary
failures (was: Hints for bootstrapping nodes are dropped under temporary
failures)
> Token metadata for bootstrapping nodes is lost under temporary failures
> ---
>
> Key: CASSANDRA-15439
> URL: https://issues.apache.org/jira/browse/CASSANDRA-15439
> Project: Cassandra
> Issue Type: Bug
>Reporter: Josh Snyder
>Priority: Normal
>
> In CASSANDRA-8838, [~pauloricardomg] asked "hints will not be stored to the
> bootstrapping node after RING_DELAY, since it will evicted from the TMD
> pending ranges. Should we create a ticket to address this?"
> CASSANDRA-15264 relates to the most likely cause of such situations, where
> the Cassandra daemon on the bootstrapping node completely crashes. Based on
> testing with {{kill -STOP}} on a bootstrapping Cassandra JVM, I believe it
> also is possible to remove token metadata (and thus pending ranges, and thus
> hints) for a bootstrapping node, simply by affecting its status in the
> failure detector.
> A node in the cluster sees the bootstrapping node this way:
> {noformat}
> INFO [GossipStage:1] 2019-11-27 20:41:41,101 Gossiper.java: - Node
> /PUBLIC-IP is now part of the cluster
> INFO [GossipStage:1] 2019-11-27 20:41:41,199 Gossiper.java:1073 -
> InetAddress /PUBLIC-IP is now UP
> INFO [HANDSHAKE-/PRIVATE-IP] 2019-11-27 20:41:41,412
> OutboundTcpConnection.java:565 - Handshaking version with /PRIVATE-IP
> INFO [STREAM-INIT-/PRIVATE-IP:21233] 2019-11-27 20:42:10,019
> StreamResultFuture.java:112 - [Stream #6219a950-1156-11ea-b45d-4d30364576c4
> ID#0] Creating new streaming plan for Bootstrap
> INFO [STREAM-INIT-/PRIVATE-IP:21233] 2019-11-27 20:42:10,020
> StreamResultFuture.java:119 - [Stream #6219a950-1156-11ea-b45d-4d30364576c4,
> ID#0] Received streaming plan for Bootstrap
> INFO [STREAM-INIT-/PRIVATE-IP:56003] 2019-11-27 20:42:10,112
> StreamResultFuture.java:119 - [Stream #6219a950-1156-11ea-b45d-4d30364576c4,
> ID#0] Received streaming plan for Bootstrap
> INFO [STREAM-IN-/PUBLIC-IP] 2019-11-27 20:42:10,179
> StreamResultFuture.java:169 - [Stream #6219a950-1156-11ea-b45d-4d30364576c4
> ID#0] Prepare completed. Receiving 0 files(0 bytes), sending 833
> files(139744616815 bytes)
> INFO [GossipStage:1] 2019-11-27 20:54:47,547 Gossiper.java:1089 -
> InetAddress /PUBLIC-IP is now DOWN
> INFO [GossipTasks:1] 2019-11-27 20:54:57,551 Gossiper.java:849 - FatClient
> /PUBLIC-IP has been silent for 3ms, removing from gossip
> {noformat}
> Since the bootstrapping node has no tokens, it is treated like a fat client,
> and it is removed from the ring. For correctness purposes, I believe we must
> keep storing hints for the downed bootstrapping node until it is either
> assassinated or until a replacement attempts to bootstrap for the same token.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org
