[
https://issues.apache.org/jira/browse/CASSANDRA-16264?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Yongle Zhang updated CASSANDRA-16264:
-------------------------------------
Description:
Steps to reproduce:
# Start a 2-node C* 2.0.0 cluster with one seed
# upgrade the non-seed node to 2.1.0
Error stacktrace on upgraded node:
{code:java}
ERROR [main] 2020-07-22 08:12:53,406 CassandraDaemon.java:474 - Exception
encountered during startup
java.lang.RuntimeException: Unable to gossip with any seeds
at org.apache.cassandra.gms.Gossiper.doShadowRound(Gossiper.java:1200)
~[apache-cassandra-2.1.0.jar:2.1.0]
at
org.apache.cassandra.service.StorageService.checkForEndpointCollision(StorageService.java:451)
~[apache-cassandra-2.1.0.jar:2.1.0]
at
org.apache.cassandra.service.StorageService.prepareToJoin(StorageService.java:667)
~[apache-cassandra-2.1.0.jar:2.1.0]
at
org.apache.cassandra.service.StorageService.initServer(StorageService.java:615)
~[apache-cassandra-2.1.0.jar:2.1.0]
at
org.apache.cassandra.service.StorageService.initServer(StorageService.java:509)
~[apache-cassandra-2.1.0.jar:2.1.0]
at
org.apache.cassandra.service.CassandraDaemon.setup(CassandraDaemon.java:338)
[apache-cassandra-2.1.0.jar:2.1.0]
at
org.apache.cassandra.service.CassandraDaemon.activate(CassandraDaemon.java:457)
[apache-cassandra-2.1.0.jar:2.1.0]
at
org.apache.cassandra.service.CassandraDaemon.main(CassandraDaemon.java:546)
[apache-cassandra-2.1.0.jar:2.1.0]
{code}
Analysis:
When a 2.1.0 node tries to make a connection with a 2.0.0 node, the 2.0.0 node
will send an integer representing its version. After receiving this integer,
the 2.1.0 node will compare it to the guessed version, and will update the
guessed version and shutdown the connection if it is older than the guessed
version, then it will make a second trial to connect to the 2.0.0 node.
However, the 2.1.0 node somehow got stuck at
https://github.com/apache/cassandra/blob/c6a2c65a75adea9a62896269da98dd036c8e57f3/src/java/org/apache/cassandra/net/OutboundTcpConnection.java#L138
and was never able to make the second connection trial. And thus the failure.
was:
Steps to reproduce:
# Start a 2-node C* 2.0.0 cluster with one seed
# upgrade the non-seed node to 2.1.0
Error stacktrace on upgraded node:
{code:java}
ERROR [main] 2020-07-22 08:12:53,406 CassandraDaemon.java:474 - Exception
encountered during startup
java.lang.RuntimeException: Unable to gossip with any seeds
at org.apache.cassandra.gms.Gossiper.doShadowRound(Gossiper.java:1200)
~[apache-cassandra-2.1.0.jar:2.1.0]
at
org.apache.cassandra.service.StorageService.checkForEndpointCollision(StorageService.java:451)
~[apache-cassandra-2.1.0.jar:2.1.0]
at
org.apache.cassandra.service.StorageService.prepareToJoin(StorageService.java:667)
~[apache-cassandra-2.1.0.jar:2.1.0]
at
org.apache.cassandra.service.StorageService.initServer(StorageService.java:615)
~[apache-cassandra-2.1.0.jar:2.1.0]
at
org.apache.cassandra.service.StorageService.initServer(StorageService.java:509)
~[apache-cassandra-2.1.0.jar:2.1.0]
at
org.apache.cassandra.service.CassandraDaemon.setup(CassandraDaemon.java:338)
[apache-cassandra-2.1.0.jar:2.1.0]
at
org.apache.cassandra.service.CassandraDaemon.activate(CassandraDaemon.java:457)
[apache-cassandra-2.1.0.jar:2.1.0]
at
org.apache.cassandra.service.CassandraDaemon.main(CassandraDaemon.java:546)
[apache-cassandra-2.1.0.jar:2.1.0]
{code}
Analysis:
When a 2.1.0 node tries to make a connection with a 2.0.0 node, the 2.0.0 node
will send an integer representing its version. After receiving this integer,
the 2.1.0 node will compare it to the guessed version, and will update the
guessed version and shutdown the connection if it is older than the guessed
version, then it will make a second trial to connect to the 2.0.0 node.
> During rolling upgrade from C* 2.0.0 to 2.1.0, upgraded node fails to gossip
> with seed
> --------------------------------------------------------------------------------------
>
> Key: CASSANDRA-16264
> URL: https://issues.apache.org/jira/browse/CASSANDRA-16264
> Project: Cassandra
> Issue Type: Bug
> Reporter: Yongle Zhang
> Priority: Normal
>
> Steps to reproduce:
> # Start a 2-node C* 2.0.0 cluster with one seed
> # upgrade the non-seed node to 2.1.0
> Error stacktrace on upgraded node:
>
> {code:java}
> ERROR [main] 2020-07-22 08:12:53,406 CassandraDaemon.java:474 - Exception
> encountered during startup
> java.lang.RuntimeException: Unable to gossip with any seeds
> at org.apache.cassandra.gms.Gossiper.doShadowRound(Gossiper.java:1200)
> ~[apache-cassandra-2.1.0.jar:2.1.0]
> at
> org.apache.cassandra.service.StorageService.checkForEndpointCollision(StorageService.java:451)
> ~[apache-cassandra-2.1.0.jar:2.1.0]
> at
> org.apache.cassandra.service.StorageService.prepareToJoin(StorageService.java:667)
> ~[apache-cassandra-2.1.0.jar:2.1.0]
> at
> org.apache.cassandra.service.StorageService.initServer(StorageService.java:615)
> ~[apache-cassandra-2.1.0.jar:2.1.0]
> at
> org.apache.cassandra.service.StorageService.initServer(StorageService.java:509)
> ~[apache-cassandra-2.1.0.jar:2.1.0]
> at
> org.apache.cassandra.service.CassandraDaemon.setup(CassandraDaemon.java:338)
> [apache-cassandra-2.1.0.jar:2.1.0]
> at
> org.apache.cassandra.service.CassandraDaemon.activate(CassandraDaemon.java:457)
> [apache-cassandra-2.1.0.jar:2.1.0]
> at
> org.apache.cassandra.service.CassandraDaemon.main(CassandraDaemon.java:546)
> [apache-cassandra-2.1.0.jar:2.1.0]
> {code}
> Analysis:
> When a 2.1.0 node tries to make a connection with a 2.0.0 node, the 2.0.0
> node will send an integer representing its version. After receiving this
> integer, the 2.1.0 node will compare it to the guessed version, and will
> update the guessed version and shutdown the connection if it is older than
> the guessed version, then it will make a second trial to connect to the 2.0.0
> node.
> However, the 2.1.0 node somehow got stuck at
> https://github.com/apache/cassandra/blob/c6a2c65a75adea9a62896269da98dd036c8e57f3/src/java/org/apache/cassandra/net/OutboundTcpConnection.java#L138
>
> and was never able to make the second connection trial. And thus the failure.
>
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
