[
https://issues.apache.org/jira/browse/CASSANDRA-9136?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14488057#comment-14488057
]
Sylvain Lebresne commented on CASSANDRA-9136:
---------------------------------------------
Am I missing something or this is not at all related to mixed cluster/upgrade?
It's just a case of querying a table before schema agreement has been reached,
which well, you shouldn't do (and we explicitly let that be the client job).
bq. According to CASSANDRA-5275, it's expected behavior to close the connection
when a request is made for an unrecognized table.
Are you sure that was the ticket you meant to link to? That doesn't seem
related to closing connections or not from what I can tell.
bq. In 2.0 and 2.1, we could read the remainder of the message, log at WARN,
and drop the message.
At least for 2.1, I don't think that's trivial because we depend on the
CFMetaData to deserialize the remainder of the message. In theory, that
dependence is not terribly strong and it might be possible to write methods to
skip the remainder even if we don't know the table, but it's not _that_
straightforward. I would suspect that's also not the only message that can have
that problem so I would have a preference for a slightly more general solution.
In practice, we do ship the payload size with messages, so in theory it
shouldn't be too hard to generally skip to the end of the message on error (at
least on that kind of error where there is no reason to suspect the connection
is screwed), which could be an option. That said, I don't really mind if you
prefer the previous option.
bq. In 3.0, we could use the new Read/WriteFailureException hooks to signal a
failure.
Yes, though we still have to make sure we deserialize the remainder of the
message (or skip it somehow).
> Mixed 2.0.14 - 2.1.4 Cluster Error Deserializing RangeSliceCommand
> ------------------------------------------------------------------
>
> Key: CASSANDRA-9136
> URL: https://issues.apache.org/jira/browse/CASSANDRA-9136
> Project: Cassandra
> Issue Type: Bug
> Components: Core
> Environment: 3 Nodes GCE, N1-Standard-2, Ubuntu 12, 1 Node on 2.1.4,
> 2 on 2.0.14
> Reporter: Russell Alexander Spitzer
> Assignee: Tyler Hobbs
> Fix For: 2.1.5
>
>
> This error occurs during a rolling upgrade between 2.0.14 and 2.1.4.
> h3. Repo
> With all the nodes on 2.0.14 make the following tables
> {code}
> CREATE KEYSPACE test WITH replication = {
> 'class': 'SimpleStrategy',
> 'replication_factor': '2'
> };
> USE test;
> CREATE TABLE compact (
> k int,
> c int,
> d int,
> PRIMARY KEY ((k), c)
> ) WITH COMPACT STORAGE;
> CREATE TABLE norm (
> k int,
> c int,
> d int,
> PRIMARY KEY ((k), c)
> ) ;
> {code}
> Then load some data into these tables. I used the python driver
> {code}
> from cassandra.cluster import Cluster
> s = Cluster().connect()
> for x in range (1000):
> for y in range (1000):
> s.execute_async("INSERT INTO test.compact (k,c,d) VALUES
> (%d,%d,%d)"%(x,y,y))
> s.execute_async("INSERT INTO test.norm (k,c,d) VALUES
> (%d,%d,%d)"%(x,y,y))
> {code}
> Upgrade one node from 2.0.14 -> 2.1.4
> From the 2.1.4 node, create a new table.
> Query that table
> On the 2.0.14 nodes you get these exceptions because the schema didn't
> propagate there. This exception kills the TCP connection between the nodes.
> {code}
> ERROR [Thread-19] 2015-04-08 18:48:45,337 CassandraDaemon.java (line 258)
> Exception in thread Thread[Thread-19,5,main]
> java.lang.NullPointerException
> at
> org.apache.cassandra.db.RangeSliceCommandSerializer.deserialize(RangeSliceCommand.java:247)
> at
> org.apache.cassandra.db.RangeSliceCommandSerializer.deserialize(RangeSliceCommand.java:156)
> at org.apache.cassandra.net.MessageIn.read(MessageIn.java:99)
> at
> org.apache.cassandra.net.IncomingTcpConnection.receiveMessage(IncomingTcpConnection.java:149)
> at
> org.apache.cassandra.net.IncomingTcpConnection.receiveMessages(IncomingTcpConnection.java:131)
> at
> org.apache.cassandra.net.IncomingTcpConnection.run(IncomingTcpConnection.java:74)
> {code}
> Run cqlsh on the upgraded node and queries will fail until the TCP connection
> is established again, easiest to repo with CL = ALL
> {code}
> cqlsh> SELECT count(*) FROM test.norm where k = 22 ;
> ReadTimeout: code=1200 [Coordinator node timed out waiting for replica nodes'
> responses] message="Operation timed out - received only 1 responses."
> info={'received_responses': 1, 'required_responses': 2, 'consistency': 'ALL'}
> cqlsh> SELECT count(*) FROM test.norm where k = 21 ;
> ReadTimeout: code=1200 [Coordinator node timed out waiting for replica nodes'
> responses] message="Operation timed out - received only 1 responses."
> info={'received_responses': 1, 'required_responses': 2, 'consistency': 'ALL'}
> {code}
> So connection made:
> {code}
> DEBUG [Thread-227] 2015-04-09 05:09:02,718 IncomingTcpConnection.java (line
> 107) Set version for /10.240.14.115 to 8 (will use 7)
> {code}
> Connection broken by query of table before schema propagated:
> {code}
> ERROR [Thread-227] 2015-04-09 05:10:24,015 CassandraDaemon.java (line 258)
> Exception in thread Thread[Thread-227,5,main]
> java.lang.NullPointerException
> at
> org.apache.cassandra.db.RangeSliceCommandSerializer.deserialize(RangeSliceCommand.java:247)
> at
> org.apache.cassandra.db.RangeSliceCommandSerializer.deserialize(RangeSliceCommand.java:156)
> at org.apache.cassandra.net.MessageIn.read(MessageIn.java:99)
> at
> org.apache.cassandra.net.IncomingTcpConnection.receiveMessage(IncomingTcpConnection.java:149)
> at
> org.apache.cassandra.net.IncomingTcpConnection.receiveMessages(IncomingTcpConnection.java:131)
> at
> org.apache.cassandra.net.IncomingTcpConnection.run(IncomingTcpConnection.java:74)
> {code}
> All query to that node will fail with timeouts now until...
> Connection re-established
> {code}
> DEBUG [Thread-228] 2015-04-09 05:11:00,323 IncomingTcpConnection.java (line
> 107) Set version for /10.240.14.115 to 8 (will use 7)
> {code}
> Now queries work again.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
