Anthony Cozzie created CASSANDRA-11403:
------------------------------------------
Summary: Serializer/Version mismatch during upgrades to C* 3.0
Key: CASSANDRA-11403
URL: https://issues.apache.org/jira/browse/CASSANDRA-11403
Project: Cassandra
Issue Type: Bug
Components: Streaming and Messaging
Reporter: Anthony Cozzie
The problem line seems to be:
{code}
MessageOut<ReadCommand> message =
readCommand.createMessage(MessagingService.instance().getVersion(endpoint));
{code}
SinglePartitionReadCommand then picks the serializer based on the version:
{code}
return new MessageOut<>(MessagingService.Verb.READ, this, version <
MessagingService.VERSION_30 ? legacyReadCommandSerializer : serializer);
{code}
However, OutboundTcpConnectionPool will test the payload size vs the version
from its smallMessages connection:
{code}
return msg.payloadSize(smallMessages.getTargetVersion()) >
LARGE_MESSAGE_THRESHOLD
{code}
Which is set when the connection/pool is created:
{code}
targetVersion = MessagingService.instance().getVersion(pool.endPoint());
{code}
During an upgrade, this state can change between these two calls leading the
3.0 serializer being used on 2.x packets and the following stacktrace:
ERROR [OptionalTasks:1] 2016-03-07 19:53:06,445 CassandraDaemon.java:195 -
Exception in thread Thread[OptionalTasks:1,5,main]
java.lang.AssertionError: null
at
org.apache.cassandra.db.ReadCommand$Serializer.serializedSize(ReadCommand.java:632)
~[cassandra-all-3.0.3.903.jar:3.0.3.903]
at
org.apache.cassandra.db.ReadCommand$Serializer.serializedSize(ReadCommand.java:536)
~[cassandra-all-3.0.3.903.jar:3.0.3.903]
at org.apache.cassandra.net.MessageOut.payloadSize(MessageOut.java:166)
~[cassandra-all-3.0.3.903.jar:3.0.3.903]
at
org.apache.cassandra.net.OutboundTcpConnectionPool.getConnection(OutboundTcpConnectionPool.java:72)
~[cassandra-all-3.0.3.903.jar:3.0.3.903]
at
org.apache.cassandra.net.MessagingService.getConnection(MessagingService.java:609)
~[cassandra-all-3.0.3.903.jar:3.0.3.903]
at
org.apache.cassandra.net.MessagingService.sendOneWay(MessagingService.java:758)
~[cassandra-all-3.0.3.903.jar:3.0.3.903]
at
org.apache.cassandra.net.MessagingService.sendRR(MessagingService.java:701)
~[cassandra-all-3.0.3.903.jar:3.0.3.903]
at
org.apache.cassandra.net.MessagingService.sendRRWithFailure(MessagingService.java:684)
~[cassandra-all-3.0.3.903.jar:3.0.3.903]
at
org.apache.cassandra.service.AbstractReadExecutor.makeRequests(AbstractReadExecutor.java:110)
~[cassandra-all-3.0.3.903.jar:3.0.3.903]
at
org.apache.cassandra.service.AbstractReadExecutor.makeDataRequests(AbstractReadExecutor.java:85)
~[cassandra-all-3.0.3.903.jar:3.0.3.903]
at
org.apache.cassandra.service.AbstractReadExecutor$NeverSpeculatingReadExecutor.executeAsync(AbstractReadExecutor.java:214)
~[cassandra-all-3.0.3.903.jar:3.0.3.903]
at
org.apache.cassandra.service.StorageProxy$SinglePartitionReadLifecycle.doInitialQueries(StorageProxy.java:1699)
~[cassandra-all-3.0.3.903.jar:3.0.3.903]
at
org.apache.cassandra.service.StorageProxy.fetchRows(StorageProxy.java:1654)
~[cassandra-all-3.0.3.903.jar:3.0.3.903]
at
org.apache.cassandra.service.StorageProxy.readRegular(StorageProxy.java:1601)
~[cassandra-all-3.0.3.903.jar:3.0.3.903]
at
org.apache.cassandra.service.StorageProxy.read(StorageProxy.java:1520)
~[cassandra-all-3.0.3.903.jar:3.0.3.903]
at
org.apache.cassandra.db.SinglePartitionReadCommand$Group.execute(SinglePartitionReadCommand.java:918)
~[cassandra-all-3.0.3.903.jar:3.0.3.903]
at
org.apache.cassandra.cql3.statements.SelectStatement.execute(SelectStatement.java:251)
~[cassandra-all-3.0.3.903.jar:3.0.3.903]
at
org.apache.cassandra.cql3.statements.SelectStatement.execute(SelectStatement.java:212)
~[cassandra-all-3.0.3.903.jar:3.0.3.903]
at
org.apache.cassandra.cql3.statements.SelectStatement.execute(SelectStatement.java:77)
~[cassandra-all-3.0.3.903.jar:3.0.3.903]
at
org.apache.cassandra.cql3.QueryProcessor.processStatement(QueryProcessor.java:206)
~[cassandra-all-3.0.3.903.jar:3.0.3.903]
at
org.apache.cassandra.cql3.QueryProcessor.process(QueryProcessor.java:237)
~[cassandra-all-3.0.3.903.jar:3.0.3.903]
at
org.apache.cassandra.cql3.QueryProcessor.process(QueryProcessor.java:252)
~[cassandra-all-3.0.3.903.jar:3.0.3.903]
at
org.apache.cassandra.cql3.QueryProcessor.process(QueryProcessor.java:247)
~[cassandra-all-3.0.3.903.jar:3.0.3.903]
at
org.apache.cassandra.auth.CassandraRoleManager.hasExistingRoles(CassandraRoleManager.java:354)
~[cassandra-all-3.0.3.903.jar:3.0.3.903]
at
org.apache.cassandra.auth.CassandraRoleManager.setupDefaultRole(CassandraRoleManager.java:330)
~[cassandra-all-3.0.3.903.jar:3.0.3.903]
at
org.apache.cassandra.auth.CassandraRoleManager.lambda$setup$148(CassandraRoleManager.java:165)
~[cassandra-all-3.0.3.903.jar:3.0.3.903]
at
org.apache.cassandra.auth.CassandraRoleManager$$Lambda$206/135350257.call(Unknown
Source) ~[na:na]
at
org.apache.cassandra.auth.CassandraRoleManager$4.run(CassandraRoleManager.java:379)
~[cassandra-all-3.0.3.903.jar:3.0.3.903]
at
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
~[na:1.8.0_40]
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
~[na:1.8.0_40]
at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180)
~[na:1.8.0_40]
at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
~[na:1.8.0_40]
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
~[na:1.8.0_40]
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
[na:1.8.0_40]
at java.lang.Thread.run(Thread.java:745) [na:1.8.0_40]
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
