Yongle Zhang created CASSANDRA-16269:
----------------------------------------
Summary: Stress testing a mixed cluster with C* 1.2.0 (seed) and
1.1.0 fails with NumberFormatException
Key: CASSANDRA-16269
URL: https://issues.apache.org/jira/browse/CASSANDRA-16269
Project: Cassandra
Issue Type: Bug
Reporter: Yongle Zhang
Steps to reproduce:
# Set up a mixed 1.1.0 & 1.2.0 cluster with 2 nodes (1 seed, 1.2.0 node being
the seed).
# Run the stress testing tool, e.g., `/cassandra/tools/bin/stress -d
250.16.238.1,250.16.238.2 -r`, wait for finish.
Error log:
{code:java}
DEBUG [WRITE-/250.16.238.1] 2020-06-18 01:55:09,606 OutboundTcpConnection.java
(line 229) attempting to connect to /250.16.238.1
DEBUG [InternalResponseStage:1] 2020-06-18 01:55:09,613
ResponseVerbHandler.java (line 44) Processing response on a callback from
122@/250.16.238.1
ERROR [InternalResponseStage:1] 2020-06-18 01:55:09,615
AbstractCassandraDaemon.java (line 134) Exception in thread
Thread[InternalResponseStage:1,5,main]
java.lang.NumberFormatException: For input string: "'115"
at
java.lang.NumberFormatException.forInputString(NumberFormatException.java:65)
at java.lang.Integer.parseInt(Integer.java:569)
at java.math.BigInteger.<init>(BigInteger.java:470)
at java.math.BigInteger.<init>(BigInteger.java:606)
at
org.apache.cassandra.dht.RandomPartitioner$1.fromString(RandomPartitioner.java:136)
at
org.apache.cassandra.dht.BootStrapper$BootstrapTokenCallback.response(BootStrapper.java:213)
at
org.apache.cassandra.net.ResponseVerbHandler.doVerb(ResponseVerbHandler.java:45)
at
org.apache.cassandra.net.MessageDeliveryTask.run(MessageDeliveryTask.java:59)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
DEBUG [InternalResponseStage:2] 2020-06-18 01:55:39,610
ResponseVerbHandler.java (line 44) Processing response on a callback from
262@/250.16.238.1
ERROR [InternalResponseStage:2] 2020-06-18 01:55:39,611
AbstractCassandraDaemon.java (line 134) Exception in thread
Thread[InternalResponseStage:2,5,main]
java.lang.NumberFormatException: For input string: "^@'115"
at
java.lang.NumberFormatException.forInputString(NumberFormatException.java:65)
at java.lang.Integer.parseInt(Integer.java:569)
at java.math.BigInteger.<init>(BigInteger.java:470)
at java.math.BigInteger.<init>(BigInteger.java:606)
at
org.apache.cassandra.dht.RandomPartitioner$1.fromString(RandomPartitioner.java:136)
at
org.apache.cassandra.dht.BootStrapper$BootstrapTokenCallback.response(BootStrapper.java:213)
at
org.apache.cassandra.net.ResponseVerbHandler.doVerb(ResponseVerbHandler.java:45)
at
org.apache.cassandra.net.MessageDeliveryTask.run(MessageDeliveryTask.java:59)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)ERROR [main] 2020-06-18 01:57:39,614
AbstractCassandraDaemon.java (line 370) Exception encountered during startup
java.lang.RuntimeException: Bootstrap failed, could not obtain token from:
/250.16.238.1
at
org.apache.cassandra.dht.BootStrapper.getBootstrapTokenFrom(BootStrapper.java:177)
at
org.apache.cassandra.dht.BootStrapper.getBalancedToken(BootStrapper.java:110)
at
org.apache.cassandra.dht.BootStrapper.getBootstrapToken(BootStrapper.java:104)
at
org.apache.cassandra.service.StorageService.joinTokenRing(StorageService.java:599)
at
org.apache.cassandra.service.StorageService.initServer(StorageService.java:515)
at
org.apache.cassandra.service.StorageService.initServer(StorageService.java:407)
at
org.apache.cassandra.service.AbstractCassandraDaemon.setup(AbstractCassandraDaemon.java:231)
at
org.apache.cassandra.service.AbstractCassandraDaemon.activate(AbstractCassandraDaemon.java:353)
at org.apache.cassandra.thrift.CassandraDaemon.main(CassandraDaemon.java:106)
ERROR [main] 2020-06-18 01:58:13,206 AbstractCassandraDaemon.java (line 370)
Exception encountered during startup
java.lang.RuntimeException: No other nodes seen! Unable to bootstrap.If you
intended to start a single-node cluster, you should make sure your
broadcast_address (or listen_address) is listed as a seed. Otherwise, you need
to determine why the seed being contacted has no knowledge of the rest of the
cluster. Usually, this can be solved by giving all nodes the same seed list.
at
org.apache.cassandra.dht.BootStrapper.getBootstrapSource(BootStrapper.java:127)
at
org.apache.cassandra.dht.BootStrapper.getBalancedToken(BootStrapper.java:109)
at
org.apache.cassandra.dht.BootStrapper.getBootstrapToken(BootStrapper.java:104)
at
org.apache.cassandra.service.StorageService.joinTokenRing(StorageService.java:599)
at
org.apache.cassandra.service.StorageService.initServer(StorageService.java:515)
at
org.apache.cassandra.service.StorageService.initServer(StorageService.java:407)
at
org.apache.cassandra.service.AbstractCassandraDaemon.setup(AbstractCassandraDaemon.java:231)
at
org.apache.cassandra.service.AbstractCassandraDaemon.activate(AbstractCassandraDaemon.java:353)
at org.apache.cassandra.thrift.CassandraDaemon.main(CassandraDaemon.java:106)
INFO [StorageServiceShutdownHook] 2020-06-18 01:58:13,213
MessagingService.java (line 539) Waiting for messaging service to quiesce
INFO [ACCEPT-/250.16.238.3] 2020-06-18 01:58:13,215 MessagingService.java
(line 695) MessagingService shutting down server thread.
DEBUG [GossipTasks:1] 2020-06-18 01:58:13,244 DebuggableThreadPoolExecutor.java
(line 167) Task cancelled
java.util.concurrent.CancellationException
at java.util.concurrent.FutureTask.report(FutureTask.java:121)
at java.util.concurrent.FutureTask.get(FutureTask.java:192)
at
org.apache.cassandra.concurrent.DebuggableThreadPoolExecutor.extractThrowable(DebuggableThreadPoolExecutor.java:159)
at
org.apache.cassandra.concurrent.DebuggableThreadPoolExecutor.logExceptionsAfterExecute(DebuggableThreadPoolExecutor.java:123)
at
org.apache.cassandra.concurrent.DebuggableScheduledThreadPoolExecutor.afterExecute(DebuggableScheduledThreadPoolExecutor.java:50)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1157)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
{code}
Root cause analysis:
In 1.2.0, when a `MessageOut<T>` class, where `T` is the type of its payload,
is serialized, it will first write the logical header part of it (which is
compatible with 1.1.0), and then write a int to indicate the length of its
payload, and then serialize its payload. When `T` is `String`, it will call
`DataOutput.writeUTF`, which will write a short to indicate the length of the
string.
In 1.1.0, the incoming byte stream will be serialized as `Message` class, which
will first serialize the header part, and then read a int as the length of the
following payload, and then read the following raw bytes as its payload. The
problem is that the payload will then be directly encoded as a UTF string, so
*the short will be considered as part of the string*. For example, this log
shows:
{code:java}
WARN [InternalResponseStage:3] 2020-07-10 08:14:08,903 RandomPartitioner.java
(line 145) Token.TokenFactory<BigInteger>.fromString: string =
�'163345392646151285100494393711260197750, bytes = 0 39 49 54 51 51 52 53 51 57
50 54 52 54 49 53 49 50 56 53 49 48 48 52 57 52 51 57 51 55 49 49 50 54 48 49
57 55 55 53 48
{code}
The first two byte 0 39 is the big endian short representation of number 39,
which is exactly the length of the following string. These two bytes can’t be
parsed as a number, so that a NumberFormatException happens.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
