[jira] [Commented] (CASSANDRA-11403) Serializer/Version mismatch during upgrades to C* 3.0
[
https://issues.apache.org/jira/browse/CASSANDRA-11403?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15370947#comment-15370947
]
Anthony Cozzie commented on CASSANDRA-11403:
I skimmed the code for CASSANDRA-11393, and it seems to be doing exactly what I
was planning to do, i.e. using a single serializer that forwards requests to
3.0 or legacy serializers. That should fix the race condition. I went ahead
and marked this as a duplicate.
> Serializer/Version mismatch during upgrades to C* 3.0
> -
>
> Key: CASSANDRA-11403
> URL: https://issues.apache.org/jira/browse/CASSANDRA-11403
> Project: Cassandra
> Issue Type: Bug
> Components: Streaming and Messaging
>Reporter: Anthony Cozzie
>
> The problem line seems to be:
> {code}
> MessageOut message =
> readCommand.createMessage(MessagingService.instance().getVersion(endpoint));
> {code}
> SinglePartitionReadCommand then picks the serializer based on the version:
> {code}
> return new MessageOut<>(MessagingService.Verb.READ, this, version <
> MessagingService.VERSION_30 ? legacyReadCommandSerializer : serializer);
> {code}
> However, OutboundTcpConnectionPool will test the payload size vs the version
> from its smallMessages connection:
> {code}
> return msg.payloadSize(smallMessages.getTargetVersion()) >
> LARGE_MESSAGE_THRESHOLD
> {code}
> Which is set when the connection/pool is created:
> {code}
> targetVersion = MessagingService.instance().getVersion(pool.endPoint());
> {code}
> During an upgrade, this state can change between these two calls leading the
> 3.0 serializer being used on 2.x packets and the following stacktrace:
> ERROR [OptionalTasks:1] 2016-03-07 19:53:06,445 CassandraDaemon.java:195 -
> Exception in thread Thread[OptionalTasks:1,5,main]
> java.lang.AssertionError: null
> at
> org.apache.cassandra.db.ReadCommand$Serializer.serializedSize(ReadCommand.java:632)
> ~[cassandra-all-3.0.3.903.jar:3.0.3.903]
> at
> org.apache.cassandra.db.ReadCommand$Serializer.serializedSize(ReadCommand.java:536)
> ~[cassandra-all-3.0.3.903.jar:3.0.3.903]
> at org.apache.cassandra.net.MessageOut.payloadSize(MessageOut.java:166)
> ~[cassandra-all-3.0.3.903.jar:3.0.3.903]
> at
> org.apache.cassandra.net.OutboundTcpConnectionPool.getConnection(OutboundTcpConnectionPool.java:72)
> ~[cassandra-all-3.0.3.903.jar:3.0.3.903]
> at
> org.apache.cassandra.net.MessagingService.getConnection(MessagingService.java:609)
> ~[cassandra-all-3.0.3.903.jar:3.0.3.903]
> at
> org.apache.cassandra.net.MessagingService.sendOneWay(MessagingService.java:758)
> ~[cassandra-all-3.0.3.903.jar:3.0.3.903]
> at
> org.apache.cassandra.net.MessagingService.sendRR(MessagingService.java:701)
> ~[cassandra-all-3.0.3.903.jar:3.0.3.903]
> at
> org.apache.cassandra.net.MessagingService.sendRRWithFailure(MessagingService.java:684)
> ~[cassandra-all-3.0.3.903.jar:3.0.3.903]
> at
> org.apache.cassandra.service.AbstractReadExecutor.makeRequests(AbstractReadExecutor.java:110)
> ~[cassandra-all-3.0.3.903.jar:3.0.3.903]
> at
> org.apache.cassandra.service.AbstractReadExecutor.makeDataRequests(AbstractReadExecutor.java:85)
> ~[cassandra-all-3.0.3.903.jar:3.0.3.903]
> at
> org.apache.cassandra.service.AbstractReadExecutor$NeverSpeculatingReadExecutor.executeAsync(AbstractReadExecutor.java:214)
> ~[cassandra-all-3.0.3.903.jar:3.0.3.903]
> at
> org.apache.cassandra.service.StorageProxy$SinglePartitionReadLifecycle.doInitialQueries(StorageProxy.java:1699)
> ~[cassandra-all-3.0.3.903.jar:3.0.3.903]
> at
> org.apache.cassandra.service.StorageProxy.fetchRows(StorageProxy.java:1654)
> ~[cassandra-all-3.0.3.903.jar:3.0.3.903]
> at
> org.apache.cassandra.service.StorageProxy.readRegular(StorageProxy.java:1601)
> ~[cassandra-all-3.0.3.903.jar:3.0.3.903]
> at
> org.apache.cassandra.service.StorageProxy.read(StorageProxy.java:1520)
> ~[cassandra-all-3.0.3.903.jar:3.0.3.903]
> at
> org.apache.cassandra.db.SinglePartitionReadCommand$Group.execute(SinglePartitionReadCommand.java:918)
> ~[cassandra-all-3.0.3.903.jar:3.0.3.903]
> at
> org.apache.cassandra.cql3.statements.SelectStatement.execute(SelectStatement.java:251)
> ~[cassandra-all-3.0.3.903.jar:3.0.3.903]
> at
> org.apache.cassandra.cql3.statements.SelectStatement.execute(SelectStatement.java:212)
> ~[cassandra-all-3.0.3.903.jar:3.0.3.903]
> at
> org.apache.cassandra.cql3.statements.SelectStatement.execute(SelectStatement.java:77)
> ~[cassandra-all-3.0.3.903.jar:3.0.3.903]
> at
> org.apache.cassandra.cql3.QueryProcessor.processStatement(QueryProcessor.java:206)
> ~[cassandra-all-3.0.3.903.jar:3.0.3.903]
> at
> org.apache.cassandra.cql3.QueryProcessor.process(QueryProcessor.java:237)
>
[jira] [Commented] (CASSANDRA-11403) Serializer/Version mismatch during upgrades to C* 3.0
[
https://issues.apache.org/jira/browse/CASSANDRA-11403?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15370470#comment-15370470
]
Sylvain Lebresne commented on CASSANDRA-11403:
--
There is a fair chance that this is fixed by CASSANDRA-11393 patch, though if
you can reproduce this somewhat reliably and can test said patch, that would be
awesome.
> Serializer/Version mismatch during upgrades to C* 3.0
> -
>
> Key: CASSANDRA-11403
> URL: https://issues.apache.org/jira/browse/CASSANDRA-11403
> Project: Cassandra
> Issue Type: Bug
> Components: Streaming and Messaging
>Reporter: Anthony Cozzie
>
> The problem line seems to be:
> {code}
> MessageOut message =
> readCommand.createMessage(MessagingService.instance().getVersion(endpoint));
> {code}
> SinglePartitionReadCommand then picks the serializer based on the version:
> {code}
> return new MessageOut<>(MessagingService.Verb.READ, this, version <
> MessagingService.VERSION_30 ? legacyReadCommandSerializer : serializer);
> {code}
> However, OutboundTcpConnectionPool will test the payload size vs the version
> from its smallMessages connection:
> {code}
> return msg.payloadSize(smallMessages.getTargetVersion()) >
> LARGE_MESSAGE_THRESHOLD
> {code}
> Which is set when the connection/pool is created:
> {code}
> targetVersion = MessagingService.instance().getVersion(pool.endPoint());
> {code}
> During an upgrade, this state can change between these two calls leading the
> 3.0 serializer being used on 2.x packets and the following stacktrace:
> ERROR [OptionalTasks:1] 2016-03-07 19:53:06,445 CassandraDaemon.java:195 -
> Exception in thread Thread[OptionalTasks:1,5,main]
> java.lang.AssertionError: null
> at
> org.apache.cassandra.db.ReadCommand$Serializer.serializedSize(ReadCommand.java:632)
> ~[cassandra-all-3.0.3.903.jar:3.0.3.903]
> at
> org.apache.cassandra.db.ReadCommand$Serializer.serializedSize(ReadCommand.java:536)
> ~[cassandra-all-3.0.3.903.jar:3.0.3.903]
> at org.apache.cassandra.net.MessageOut.payloadSize(MessageOut.java:166)
> ~[cassandra-all-3.0.3.903.jar:3.0.3.903]
> at
> org.apache.cassandra.net.OutboundTcpConnectionPool.getConnection(OutboundTcpConnectionPool.java:72)
> ~[cassandra-all-3.0.3.903.jar:3.0.3.903]
> at
> org.apache.cassandra.net.MessagingService.getConnection(MessagingService.java:609)
> ~[cassandra-all-3.0.3.903.jar:3.0.3.903]
> at
> org.apache.cassandra.net.MessagingService.sendOneWay(MessagingService.java:758)
> ~[cassandra-all-3.0.3.903.jar:3.0.3.903]
> at
> org.apache.cassandra.net.MessagingService.sendRR(MessagingService.java:701)
> ~[cassandra-all-3.0.3.903.jar:3.0.3.903]
> at
> org.apache.cassandra.net.MessagingService.sendRRWithFailure(MessagingService.java:684)
> ~[cassandra-all-3.0.3.903.jar:3.0.3.903]
> at
> org.apache.cassandra.service.AbstractReadExecutor.makeRequests(AbstractReadExecutor.java:110)
> ~[cassandra-all-3.0.3.903.jar:3.0.3.903]
> at
> org.apache.cassandra.service.AbstractReadExecutor.makeDataRequests(AbstractReadExecutor.java:85)
> ~[cassandra-all-3.0.3.903.jar:3.0.3.903]
> at
> org.apache.cassandra.service.AbstractReadExecutor$NeverSpeculatingReadExecutor.executeAsync(AbstractReadExecutor.java:214)
> ~[cassandra-all-3.0.3.903.jar:3.0.3.903]
> at
> org.apache.cassandra.service.StorageProxy$SinglePartitionReadLifecycle.doInitialQueries(StorageProxy.java:1699)
> ~[cassandra-all-3.0.3.903.jar:3.0.3.903]
> at
> org.apache.cassandra.service.StorageProxy.fetchRows(StorageProxy.java:1654)
> ~[cassandra-all-3.0.3.903.jar:3.0.3.903]
> at
> org.apache.cassandra.service.StorageProxy.readRegular(StorageProxy.java:1601)
> ~[cassandra-all-3.0.3.903.jar:3.0.3.903]
> at
> org.apache.cassandra.service.StorageProxy.read(StorageProxy.java:1520)
> ~[cassandra-all-3.0.3.903.jar:3.0.3.903]
> at
> org.apache.cassandra.db.SinglePartitionReadCommand$Group.execute(SinglePartitionReadCommand.java:918)
> ~[cassandra-all-3.0.3.903.jar:3.0.3.903]
> at
> org.apache.cassandra.cql3.statements.SelectStatement.execute(SelectStatement.java:251)
> ~[cassandra-all-3.0.3.903.jar:3.0.3.903]
> at
> org.apache.cassandra.cql3.statements.SelectStatement.execute(SelectStatement.java:212)
> ~[cassandra-all-3.0.3.903.jar:3.0.3.903]
> at
> org.apache.cassandra.cql3.statements.SelectStatement.execute(SelectStatement.java:77)
> ~[cassandra-all-3.0.3.903.jar:3.0.3.903]
> at
> org.apache.cassandra.cql3.QueryProcessor.processStatement(QueryProcessor.java:206)
> ~[cassandra-all-3.0.3.903.jar:3.0.3.903]
> at
> org.apache.cassandra.cql3.QueryProcessor.process(QueryProcessor.java:237)
> ~[cassandra-all-3.0.3.903.jar:3.0.3.903]
> at
>
[jira] [Commented] (CASSANDRA-11403) Serializer/Version mismatch during upgrades to C* 3.0
[
https://issues.apache.org/jira/browse/CASSANDRA-11403?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15368524#comment-15368524
]
Jeremiah Jordan commented on CASSANDRA-11403:
-
We are seeing this sporadically on current 3.0 head still.
> Serializer/Version mismatch during upgrades to C* 3.0
> -
>
> Key: CASSANDRA-11403
> URL: https://issues.apache.org/jira/browse/CASSANDRA-11403
> Project: Cassandra
> Issue Type: Bug
> Components: Streaming and Messaging
>Reporter: Anthony Cozzie
>
> The problem line seems to be:
> {code}
> MessageOut message =
> readCommand.createMessage(MessagingService.instance().getVersion(endpoint));
> {code}
> SinglePartitionReadCommand then picks the serializer based on the version:
> {code}
> return new MessageOut<>(MessagingService.Verb.READ, this, version <
> MessagingService.VERSION_30 ? legacyReadCommandSerializer : serializer);
> {code}
> However, OutboundTcpConnectionPool will test the payload size vs the version
> from its smallMessages connection:
> {code}
> return msg.payloadSize(smallMessages.getTargetVersion()) >
> LARGE_MESSAGE_THRESHOLD
> {code}
> Which is set when the connection/pool is created:
> {code}
> targetVersion = MessagingService.instance().getVersion(pool.endPoint());
> {code}
> During an upgrade, this state can change between these two calls leading the
> 3.0 serializer being used on 2.x packets and the following stacktrace:
> ERROR [OptionalTasks:1] 2016-03-07 19:53:06,445 CassandraDaemon.java:195 -
> Exception in thread Thread[OptionalTasks:1,5,main]
> java.lang.AssertionError: null
> at
> org.apache.cassandra.db.ReadCommand$Serializer.serializedSize(ReadCommand.java:632)
> ~[cassandra-all-3.0.3.903.jar:3.0.3.903]
> at
> org.apache.cassandra.db.ReadCommand$Serializer.serializedSize(ReadCommand.java:536)
> ~[cassandra-all-3.0.3.903.jar:3.0.3.903]
> at org.apache.cassandra.net.MessageOut.payloadSize(MessageOut.java:166)
> ~[cassandra-all-3.0.3.903.jar:3.0.3.903]
> at
> org.apache.cassandra.net.OutboundTcpConnectionPool.getConnection(OutboundTcpConnectionPool.java:72)
> ~[cassandra-all-3.0.3.903.jar:3.0.3.903]
> at
> org.apache.cassandra.net.MessagingService.getConnection(MessagingService.java:609)
> ~[cassandra-all-3.0.3.903.jar:3.0.3.903]
> at
> org.apache.cassandra.net.MessagingService.sendOneWay(MessagingService.java:758)
> ~[cassandra-all-3.0.3.903.jar:3.0.3.903]
> at
> org.apache.cassandra.net.MessagingService.sendRR(MessagingService.java:701)
> ~[cassandra-all-3.0.3.903.jar:3.0.3.903]
> at
> org.apache.cassandra.net.MessagingService.sendRRWithFailure(MessagingService.java:684)
> ~[cassandra-all-3.0.3.903.jar:3.0.3.903]
> at
> org.apache.cassandra.service.AbstractReadExecutor.makeRequests(AbstractReadExecutor.java:110)
> ~[cassandra-all-3.0.3.903.jar:3.0.3.903]
> at
> org.apache.cassandra.service.AbstractReadExecutor.makeDataRequests(AbstractReadExecutor.java:85)
> ~[cassandra-all-3.0.3.903.jar:3.0.3.903]
> at
> org.apache.cassandra.service.AbstractReadExecutor$NeverSpeculatingReadExecutor.executeAsync(AbstractReadExecutor.java:214)
> ~[cassandra-all-3.0.3.903.jar:3.0.3.903]
> at
> org.apache.cassandra.service.StorageProxy$SinglePartitionReadLifecycle.doInitialQueries(StorageProxy.java:1699)
> ~[cassandra-all-3.0.3.903.jar:3.0.3.903]
> at
> org.apache.cassandra.service.StorageProxy.fetchRows(StorageProxy.java:1654)
> ~[cassandra-all-3.0.3.903.jar:3.0.3.903]
> at
> org.apache.cassandra.service.StorageProxy.readRegular(StorageProxy.java:1601)
> ~[cassandra-all-3.0.3.903.jar:3.0.3.903]
> at
> org.apache.cassandra.service.StorageProxy.read(StorageProxy.java:1520)
> ~[cassandra-all-3.0.3.903.jar:3.0.3.903]
> at
> org.apache.cassandra.db.SinglePartitionReadCommand$Group.execute(SinglePartitionReadCommand.java:918)
> ~[cassandra-all-3.0.3.903.jar:3.0.3.903]
> at
> org.apache.cassandra.cql3.statements.SelectStatement.execute(SelectStatement.java:251)
> ~[cassandra-all-3.0.3.903.jar:3.0.3.903]
> at
> org.apache.cassandra.cql3.statements.SelectStatement.execute(SelectStatement.java:212)
> ~[cassandra-all-3.0.3.903.jar:3.0.3.903]
> at
> org.apache.cassandra.cql3.statements.SelectStatement.execute(SelectStatement.java:77)
> ~[cassandra-all-3.0.3.903.jar:3.0.3.903]
> at
> org.apache.cassandra.cql3.QueryProcessor.processStatement(QueryProcessor.java:206)
> ~[cassandra-all-3.0.3.903.jar:3.0.3.903]
> at
> org.apache.cassandra.cql3.QueryProcessor.process(QueryProcessor.java:237)
> ~[cassandra-all-3.0.3.903.jar:3.0.3.903]
> at
> org.apache.cassandra.cql3.QueryProcessor.process(QueryProcessor.java:252)
> ~[cassandra-all-3.0.3.903.jar:3.0.3.903]
> at
>
[jira] [Commented] (CASSANDRA-11403) Serializer/Version mismatch during upgrades to C* 3.0
[
https://issues.apache.org/jira/browse/CASSANDRA-11403?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15214776#comment-15214776
]
Jeremiah Jordan commented on CASSANDRA-11403:
-
Closing this cannot reproduce for now. If we still see it on cassandra-3.0
head we can reopen.
> Serializer/Version mismatch during upgrades to C* 3.0
> -
>
> Key: CASSANDRA-11403
> URL: https://issues.apache.org/jira/browse/CASSANDRA-11403
> Project: Cassandra
> Issue Type: Bug
> Components: Streaming and Messaging
>Reporter: Anthony Cozzie
>
> The problem line seems to be:
> {code}
> MessageOut message =
> readCommand.createMessage(MessagingService.instance().getVersion(endpoint));
> {code}
> SinglePartitionReadCommand then picks the serializer based on the version:
> {code}
> return new MessageOut<>(MessagingService.Verb.READ, this, version <
> MessagingService.VERSION_30 ? legacyReadCommandSerializer : serializer);
> {code}
> However, OutboundTcpConnectionPool will test the payload size vs the version
> from its smallMessages connection:
> {code}
> return msg.payloadSize(smallMessages.getTargetVersion()) >
> LARGE_MESSAGE_THRESHOLD
> {code}
> Which is set when the connection/pool is created:
> {code}
> targetVersion = MessagingService.instance().getVersion(pool.endPoint());
> {code}
> During an upgrade, this state can change between these two calls leading the
> 3.0 serializer being used on 2.x packets and the following stacktrace:
> ERROR [OptionalTasks:1] 2016-03-07 19:53:06,445 CassandraDaemon.java:195 -
> Exception in thread Thread[OptionalTasks:1,5,main]
> java.lang.AssertionError: null
> at
> org.apache.cassandra.db.ReadCommand$Serializer.serializedSize(ReadCommand.java:632)
> ~[cassandra-all-3.0.3.903.jar:3.0.3.903]
> at
> org.apache.cassandra.db.ReadCommand$Serializer.serializedSize(ReadCommand.java:536)
> ~[cassandra-all-3.0.3.903.jar:3.0.3.903]
> at org.apache.cassandra.net.MessageOut.payloadSize(MessageOut.java:166)
> ~[cassandra-all-3.0.3.903.jar:3.0.3.903]
> at
> org.apache.cassandra.net.OutboundTcpConnectionPool.getConnection(OutboundTcpConnectionPool.java:72)
> ~[cassandra-all-3.0.3.903.jar:3.0.3.903]
> at
> org.apache.cassandra.net.MessagingService.getConnection(MessagingService.java:609)
> ~[cassandra-all-3.0.3.903.jar:3.0.3.903]
> at
> org.apache.cassandra.net.MessagingService.sendOneWay(MessagingService.java:758)
> ~[cassandra-all-3.0.3.903.jar:3.0.3.903]
> at
> org.apache.cassandra.net.MessagingService.sendRR(MessagingService.java:701)
> ~[cassandra-all-3.0.3.903.jar:3.0.3.903]
> at
> org.apache.cassandra.net.MessagingService.sendRRWithFailure(MessagingService.java:684)
> ~[cassandra-all-3.0.3.903.jar:3.0.3.903]
> at
> org.apache.cassandra.service.AbstractReadExecutor.makeRequests(AbstractReadExecutor.java:110)
> ~[cassandra-all-3.0.3.903.jar:3.0.3.903]
> at
> org.apache.cassandra.service.AbstractReadExecutor.makeDataRequests(AbstractReadExecutor.java:85)
> ~[cassandra-all-3.0.3.903.jar:3.0.3.903]
> at
> org.apache.cassandra.service.AbstractReadExecutor$NeverSpeculatingReadExecutor.executeAsync(AbstractReadExecutor.java:214)
> ~[cassandra-all-3.0.3.903.jar:3.0.3.903]
> at
> org.apache.cassandra.service.StorageProxy$SinglePartitionReadLifecycle.doInitialQueries(StorageProxy.java:1699)
> ~[cassandra-all-3.0.3.903.jar:3.0.3.903]
> at
> org.apache.cassandra.service.StorageProxy.fetchRows(StorageProxy.java:1654)
> ~[cassandra-all-3.0.3.903.jar:3.0.3.903]
> at
> org.apache.cassandra.service.StorageProxy.readRegular(StorageProxy.java:1601)
> ~[cassandra-all-3.0.3.903.jar:3.0.3.903]
> at
> org.apache.cassandra.service.StorageProxy.read(StorageProxy.java:1520)
> ~[cassandra-all-3.0.3.903.jar:3.0.3.903]
> at
> org.apache.cassandra.db.SinglePartitionReadCommand$Group.execute(SinglePartitionReadCommand.java:918)
> ~[cassandra-all-3.0.3.903.jar:3.0.3.903]
> at
> org.apache.cassandra.cql3.statements.SelectStatement.execute(SelectStatement.java:251)
> ~[cassandra-all-3.0.3.903.jar:3.0.3.903]
> at
> org.apache.cassandra.cql3.statements.SelectStatement.execute(SelectStatement.java:212)
> ~[cassandra-all-3.0.3.903.jar:3.0.3.903]
> at
> org.apache.cassandra.cql3.statements.SelectStatement.execute(SelectStatement.java:77)
> ~[cassandra-all-3.0.3.903.jar:3.0.3.903]
> at
> org.apache.cassandra.cql3.QueryProcessor.processStatement(QueryProcessor.java:206)
> ~[cassandra-all-3.0.3.903.jar:3.0.3.903]
> at
> org.apache.cassandra.cql3.QueryProcessor.process(QueryProcessor.java:237)
> ~[cassandra-all-3.0.3.903.jar:3.0.3.903]
> at
> org.apache.cassandra.cql3.QueryProcessor.process(QueryProcessor.java:252)
> ~[cassandra-all-3.0.3.903.jar:3.0.3.903]
>
[jira] [Commented] (CASSANDRA-11403) Serializer/Version mismatch during upgrades to C* 3.0
[
https://issues.apache.org/jira/browse/CASSANDRA-11403?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15208532#comment-15208532
]
Jeremiah Jordan commented on CASSANDRA-11403:
-
This is on cassandra-3.0 hash 9cfbc31bc29685bd60355a823e0cf261a89858f0
> Serializer/Version mismatch during upgrades to C* 3.0
> -
>
> Key: CASSANDRA-11403
> URL: https://issues.apache.org/jira/browse/CASSANDRA-11403
> Project: Cassandra
> Issue Type: Bug
> Components: Streaming and Messaging
>Reporter: Anthony Cozzie
>
> The problem line seems to be:
> {code}
> MessageOut message =
> readCommand.createMessage(MessagingService.instance().getVersion(endpoint));
> {code}
> SinglePartitionReadCommand then picks the serializer based on the version:
> {code}
> return new MessageOut<>(MessagingService.Verb.READ, this, version <
> MessagingService.VERSION_30 ? legacyReadCommandSerializer : serializer);
> {code}
> However, OutboundTcpConnectionPool will test the payload size vs the version
> from its smallMessages connection:
> {code}
> return msg.payloadSize(smallMessages.getTargetVersion()) >
> LARGE_MESSAGE_THRESHOLD
> {code}
> Which is set when the connection/pool is created:
> {code}
> targetVersion = MessagingService.instance().getVersion(pool.endPoint());
> {code}
> During an upgrade, this state can change between these two calls leading the
> 3.0 serializer being used on 2.x packets and the following stacktrace:
> ERROR [OptionalTasks:1] 2016-03-07 19:53:06,445 CassandraDaemon.java:195 -
> Exception in thread Thread[OptionalTasks:1,5,main]
> java.lang.AssertionError: null
> at
> org.apache.cassandra.db.ReadCommand$Serializer.serializedSize(ReadCommand.java:632)
> ~[cassandra-all-3.0.3.903.jar:3.0.3.903]
> at
> org.apache.cassandra.db.ReadCommand$Serializer.serializedSize(ReadCommand.java:536)
> ~[cassandra-all-3.0.3.903.jar:3.0.3.903]
> at org.apache.cassandra.net.MessageOut.payloadSize(MessageOut.java:166)
> ~[cassandra-all-3.0.3.903.jar:3.0.3.903]
> at
> org.apache.cassandra.net.OutboundTcpConnectionPool.getConnection(OutboundTcpConnectionPool.java:72)
> ~[cassandra-all-3.0.3.903.jar:3.0.3.903]
> at
> org.apache.cassandra.net.MessagingService.getConnection(MessagingService.java:609)
> ~[cassandra-all-3.0.3.903.jar:3.0.3.903]
> at
> org.apache.cassandra.net.MessagingService.sendOneWay(MessagingService.java:758)
> ~[cassandra-all-3.0.3.903.jar:3.0.3.903]
> at
> org.apache.cassandra.net.MessagingService.sendRR(MessagingService.java:701)
> ~[cassandra-all-3.0.3.903.jar:3.0.3.903]
> at
> org.apache.cassandra.net.MessagingService.sendRRWithFailure(MessagingService.java:684)
> ~[cassandra-all-3.0.3.903.jar:3.0.3.903]
> at
> org.apache.cassandra.service.AbstractReadExecutor.makeRequests(AbstractReadExecutor.java:110)
> ~[cassandra-all-3.0.3.903.jar:3.0.3.903]
> at
> org.apache.cassandra.service.AbstractReadExecutor.makeDataRequests(AbstractReadExecutor.java:85)
> ~[cassandra-all-3.0.3.903.jar:3.0.3.903]
> at
> org.apache.cassandra.service.AbstractReadExecutor$NeverSpeculatingReadExecutor.executeAsync(AbstractReadExecutor.java:214)
> ~[cassandra-all-3.0.3.903.jar:3.0.3.903]
> at
> org.apache.cassandra.service.StorageProxy$SinglePartitionReadLifecycle.doInitialQueries(StorageProxy.java:1699)
> ~[cassandra-all-3.0.3.903.jar:3.0.3.903]
> at
> org.apache.cassandra.service.StorageProxy.fetchRows(StorageProxy.java:1654)
> ~[cassandra-all-3.0.3.903.jar:3.0.3.903]
> at
> org.apache.cassandra.service.StorageProxy.readRegular(StorageProxy.java:1601)
> ~[cassandra-all-3.0.3.903.jar:3.0.3.903]
> at
> org.apache.cassandra.service.StorageProxy.read(StorageProxy.java:1520)
> ~[cassandra-all-3.0.3.903.jar:3.0.3.903]
> at
> org.apache.cassandra.db.SinglePartitionReadCommand$Group.execute(SinglePartitionReadCommand.java:918)
> ~[cassandra-all-3.0.3.903.jar:3.0.3.903]
> at
> org.apache.cassandra.cql3.statements.SelectStatement.execute(SelectStatement.java:251)
> ~[cassandra-all-3.0.3.903.jar:3.0.3.903]
> at
> org.apache.cassandra.cql3.statements.SelectStatement.execute(SelectStatement.java:212)
> ~[cassandra-all-3.0.3.903.jar:3.0.3.903]
> at
> org.apache.cassandra.cql3.statements.SelectStatement.execute(SelectStatement.java:77)
> ~[cassandra-all-3.0.3.903.jar:3.0.3.903]
> at
> org.apache.cassandra.cql3.QueryProcessor.processStatement(QueryProcessor.java:206)
> ~[cassandra-all-3.0.3.903.jar:3.0.3.903]
> at
> org.apache.cassandra.cql3.QueryProcessor.process(QueryProcessor.java:237)
> ~[cassandra-all-3.0.3.903.jar:3.0.3.903]
> at
> org.apache.cassandra.cql3.QueryProcessor.process(QueryProcessor.java:252)
> ~[cassandra-all-3.0.3.903.jar:3.0.3.903]
> at
>
[jira] [Commented] (CASSANDRA-11403) Serializer/Version mismatch during upgrades to C* 3.0
[
https://issues.apache.org/jira/browse/CASSANDRA-11403?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15208076#comment-15208076
]
Sylvain Lebresne commented on CASSANDRA-11403:
--
Which exact version of C* is that? We've fixed bugs related to this already so
this is likely a duplicate, unless you can reproduce on the current branches.
> Serializer/Version mismatch during upgrades to C* 3.0
> -
>
> Key: CASSANDRA-11403
> URL: https://issues.apache.org/jira/browse/CASSANDRA-11403
> Project: Cassandra
> Issue Type: Bug
> Components: Streaming and Messaging
>Reporter: Anthony Cozzie
>
> The problem line seems to be:
> {code}
> MessageOut message =
> readCommand.createMessage(MessagingService.instance().getVersion(endpoint));
> {code}
> SinglePartitionReadCommand then picks the serializer based on the version:
> {code}
> return new MessageOut<>(MessagingService.Verb.READ, this, version <
> MessagingService.VERSION_30 ? legacyReadCommandSerializer : serializer);
> {code}
> However, OutboundTcpConnectionPool will test the payload size vs the version
> from its smallMessages connection:
> {code}
> return msg.payloadSize(smallMessages.getTargetVersion()) >
> LARGE_MESSAGE_THRESHOLD
> {code}
> Which is set when the connection/pool is created:
> {code}
> targetVersion = MessagingService.instance().getVersion(pool.endPoint());
> {code}
> During an upgrade, this state can change between these two calls leading the
> 3.0 serializer being used on 2.x packets and the following stacktrace:
> ERROR [OptionalTasks:1] 2016-03-07 19:53:06,445 CassandraDaemon.java:195 -
> Exception in thread Thread[OptionalTasks:1,5,main]
> java.lang.AssertionError: null
> at
> org.apache.cassandra.db.ReadCommand$Serializer.serializedSize(ReadCommand.java:632)
> ~[cassandra-all-3.0.3.903.jar:3.0.3.903]
> at
> org.apache.cassandra.db.ReadCommand$Serializer.serializedSize(ReadCommand.java:536)
> ~[cassandra-all-3.0.3.903.jar:3.0.3.903]
> at org.apache.cassandra.net.MessageOut.payloadSize(MessageOut.java:166)
> ~[cassandra-all-3.0.3.903.jar:3.0.3.903]
> at
> org.apache.cassandra.net.OutboundTcpConnectionPool.getConnection(OutboundTcpConnectionPool.java:72)
> ~[cassandra-all-3.0.3.903.jar:3.0.3.903]
> at
> org.apache.cassandra.net.MessagingService.getConnection(MessagingService.java:609)
> ~[cassandra-all-3.0.3.903.jar:3.0.3.903]
> at
> org.apache.cassandra.net.MessagingService.sendOneWay(MessagingService.java:758)
> ~[cassandra-all-3.0.3.903.jar:3.0.3.903]
> at
> org.apache.cassandra.net.MessagingService.sendRR(MessagingService.java:701)
> ~[cassandra-all-3.0.3.903.jar:3.0.3.903]
> at
> org.apache.cassandra.net.MessagingService.sendRRWithFailure(MessagingService.java:684)
> ~[cassandra-all-3.0.3.903.jar:3.0.3.903]
> at
> org.apache.cassandra.service.AbstractReadExecutor.makeRequests(AbstractReadExecutor.java:110)
> ~[cassandra-all-3.0.3.903.jar:3.0.3.903]
> at
> org.apache.cassandra.service.AbstractReadExecutor.makeDataRequests(AbstractReadExecutor.java:85)
> ~[cassandra-all-3.0.3.903.jar:3.0.3.903]
> at
> org.apache.cassandra.service.AbstractReadExecutor$NeverSpeculatingReadExecutor.executeAsync(AbstractReadExecutor.java:214)
> ~[cassandra-all-3.0.3.903.jar:3.0.3.903]
> at
> org.apache.cassandra.service.StorageProxy$SinglePartitionReadLifecycle.doInitialQueries(StorageProxy.java:1699)
> ~[cassandra-all-3.0.3.903.jar:3.0.3.903]
> at
> org.apache.cassandra.service.StorageProxy.fetchRows(StorageProxy.java:1654)
> ~[cassandra-all-3.0.3.903.jar:3.0.3.903]
> at
> org.apache.cassandra.service.StorageProxy.readRegular(StorageProxy.java:1601)
> ~[cassandra-all-3.0.3.903.jar:3.0.3.903]
> at
> org.apache.cassandra.service.StorageProxy.read(StorageProxy.java:1520)
> ~[cassandra-all-3.0.3.903.jar:3.0.3.903]
> at
> org.apache.cassandra.db.SinglePartitionReadCommand$Group.execute(SinglePartitionReadCommand.java:918)
> ~[cassandra-all-3.0.3.903.jar:3.0.3.903]
> at
> org.apache.cassandra.cql3.statements.SelectStatement.execute(SelectStatement.java:251)
> ~[cassandra-all-3.0.3.903.jar:3.0.3.903]
> at
> org.apache.cassandra.cql3.statements.SelectStatement.execute(SelectStatement.java:212)
> ~[cassandra-all-3.0.3.903.jar:3.0.3.903]
> at
> org.apache.cassandra.cql3.statements.SelectStatement.execute(SelectStatement.java:77)
> ~[cassandra-all-3.0.3.903.jar:3.0.3.903]
> at
> org.apache.cassandra.cql3.QueryProcessor.processStatement(QueryProcessor.java:206)
> ~[cassandra-all-3.0.3.903.jar:3.0.3.903]
> at
> org.apache.cassandra.cql3.QueryProcessor.process(QueryProcessor.java:237)
> ~[cassandra-all-3.0.3.903.jar:3.0.3.903]
> at
>
