[
https://issues.apache.org/jira/browse/CASSANDRA-8628?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14278936#comment-14278936
]
Brandon Williams commented on CASSANDRA-8628:
---------------------------------------------
bq. My reading of the upgrading compatibility document leads me to think that
there should not be any problem in performing this kind of rebuild.
Actually, there is. Since the new node is 2.1, it won't pull schema from 2.0,
which leads to your "Couldn't find cfId" problems. You should start the new
node on 2.0, let it get the schema, then upgrade it to 2.1.
> Stalling rebuild of new 2.1.2 node from cluster running 2.0.9
> -------------------------------------------------------------
>
> Key: CASSANDRA-8628
> URL: https://issues.apache.org/jira/browse/CASSANDRA-8628
> Project: Cassandra
> Issue Type: Bug
> Components: Core
> Environment: Ubuntu 14.04, 4 core machines, 8 and 16 GB RAM
> Cluster running cassandra 2.0.9, new node running 2.1.2,
> inter-node encryption: all
> Reporter: Lennart Liberg
>
> We want to add a new datacenter to our existing cluster consisting of two
> datacenters containing 2 nodes and 1 node respectively, running version 2.0.9
> We configured a node running cassandra 2.1.2 in the new datacenter
> (auto_bootstrap: false) and verified that it managed to join the cluster as
> expected (`nodetool status` saying its status was UN).
> Upon executing `nodetool rebuild ams1`, the system.log output below is
> observed, with a new UnknownColumnFamilyException and NullPointerException
> every few minutes, and no data being transfered.
> After downgrading to cassandra 2.0.11, the rebuild operation worked
> flawlessly, using a cassandra.yaml configured as identical as possible to the
> 2.1.2 yaml file (a lot of new items in 2.1.x).
> My reading of the upgrading compatibility document leads me to think that
> there should not be any problem in performing this kind of rebuild.
> http://www.datastax.com/documentation/upgrade/doc/upgrade/cassandra/upgradeDebC_t.html
> system.log output:
> {noformat}
> INFO [RMI TCP Connection(57622)-10.240.12.239] 2015-01-15 16:00:04,228
> StorageService.java:958 - rebuild from dc: ams1
> INFO [RMI TCP Connection(57622)-10.240.12.239] 2015-01-15 16:00:04,305
> StreamResultFuture.java:86 - [Stream #91906440-9ccf-11e4-9cab-67625edb9342]
> Executing streaming plan for Rebuild
> INFO [StreamConnectionEstablisher:4] 2015-01-15 16:00:04,306
> StreamSession.java:213 - [Stream #91906440-9ccf-11e4-9cab-67625edb9342]
> Starting streaming to 37.xx.xx.1/37.xx.xx.1
> INFO [StreamConnectionEstablisher:4] 2015-01-15 16:00:04,610
> StreamCoordinator.java:209 - [Stream #91906440-9ccf-11e4-9cab-67625edb9342,
> ID#0] Beginning stream session with 37.xx.xx.1/37.xx.xx.1
> WARN [Thread-28935] 2015-01-15 16:03:40,458 IncomingTcpConnection.java:91 -
> UnknownColumnFamilyException reading from socket; closing
> org.apache.cassandra.db.UnknownColumnFamilyException: Couldn't find
> cfId=d1ea009c-feba-3d75-aa64-d022270842c4
> at
> org.apache.cassandra.db.ColumnFamilySerializer.deserializeCfId(ColumnFamilySerializer.java:164)
> ~[apache-cassandra-2.1.2.jar:2.1.2]
> at
> org.apache.cassandra.db.ColumnFamilySerializer.deserialize(ColumnFamilySerializer.java:97)
> ~[apache-cassandra-2.1.2.jar:2.1.2]
> at
> org.apache.cassandra.db.Mutation$MutationSerializer.deserializeOneCf(Mutation.java:322)
> ~[apache-cassandra-2.1.2.jar:2.1.2]
> at
> org.apache.cassandra.db.Mutation$MutationSerializer.deserialize(Mutation.java:302)
> ~[apache-cassandra-2.1.2.jar:2.1.2]
> at
> org.apache.cassandra.db.Mutation$MutationSerializer.deserialize(Mutation.java:330)
> ~[apache-cassandra-2.1.2.jar:2.1.2]
> at
> org.apache.cassandra.db.Mutation$MutationSerializer.deserialize(Mutation.java:272)
> ~[apache-cassandra-2.1.2.jar:2.1.2]
> at org.apache.cassandra.net.MessageIn.read(MessageIn.java:99)
> ~[apache-cassandra-2.1.2.jar:2.1.2]
> at
> org.apache.cassandra.net.IncomingTcpConnection.receiveMessage(IncomingTcpConnection.java:168)
> ~[apache-cassandra-2.1.2.jar:2.1.2]
> at
> org.apache.cassandra.net.IncomingTcpConnection.receiveMessages(IncomingTcpConnection.java:150)
> ~[apache-cassandra-2.1.2.jar:2.1.2]
> at
> org.apache.cassandra.net.IncomingTcpConnection.run(IncomingTcpConnection.java:82)
> ~[apache-cassandra-2.1.2.jar:2.1.2]
> ERROR [Thread-28974] 2015-01-15 16:05:54,164 CassandraDaemon.java:153 -
> Exception in thread Thread[Thread-28974,5,main]
> java.lang.NullPointerException: null
> at
> org.apache.cassandra.db.SliceFromReadCommandSerializer.deserialize(SliceFromReadCommand.java:153)
> ~[apache-cassandra-2.1.2.jar:2.1.2]
> at
> org.apache.cassandra.db.ReadCommandSerializer.deserialize(ReadCommand.java:157)
> ~[apache-cassandra-2.1.2.jar:2.1.2]
> at
> org.apache.cassandra.db.ReadCommandSerializer.deserialize(ReadCommand.java:131)
> ~[apache-cassandra-2.1.2.jar:2.1.2]
> at org.apache.cassandra.net.MessageIn.read(MessageIn.java:99)
> ~[apache-cassandra-2.1.2.jar:2.1.2]
> at
> org.apache.cassandra.net.IncomingTcpConnection.receiveMessage(IncomingTcpConnection.java:168)
> ~[apache-cassandra-2.1.2.jar:2.1.2]
> at
> org.apache.cassandra.net.IncomingTcpConnection.receiveMessages(IncomingTcpConnection.java:150)
> ~[apache-cassandra-2.1.2.jar:2.1.2]
> at
> org.apache.cassandra.net.IncomingTcpConnection.run(IncomingTcpConnection.java:82)
> ~[apache-cassandra-2.1.2.jar:2.1.2]
> ERROR [Thread-29011] 2015-01-15 16:07:58,349 CassandraDaemon.java:153 -
> Exception in thread Thread[Thread-29011,5,main]
> java.lang.NullPointerException: null
> at
> org.apache.cassandra.db.SliceFromReadCommandSerializer.deserialize(SliceFromReadCommand.java:153)
> ~[apache-cassandra-2.1.2.jar:2.1.2]
> at
> org.apache.cassandra.db.ReadCommandSerializer.deserialize(ReadCommand.java:157)
> ~[apache-cassandra-2.1.2.jar:2.1.2]
> at
> org.apache.cassandra.db.ReadCommandSerializer.deserialize(ReadCommand.java:131)
> ~[apache-cassandra-2.1.2.jar:2.1.2]
> at org.apache.cassandra.net.MessageIn.read(MessageIn.java:99)
> ~[apache-cassandra-2.1.2.jar:2.1.2]
> at
> org.apache.cassandra.net.IncomingTcpConnection.receiveMessage(IncomingTcpConnection.java:168)
> ~[apache-cassandra-2.1.2.jar:2.1.2]
> at
> org.apache.cassandra.net.IncomingTcpConnection.receiveMessages(IncomingTcpConnection.java:150)
> ~[apache-cassandra-2.1.2.jar:2.1.2]
> at
> org.apache.cassandra.net.IncomingTcpConnection.run(IncomingTcpConnection.java:82)
> ~[apache-cassandra-2.1.2.jar:2.1.2]
> ERROR [Thread-29013] 2015-01-15 16:08:01,037 CassandraDaemon.java:153 -
> Exception in thread Thread[Thread-29013,5,main]
> java.lang.NullPointerException: null
> at
> org.apache.cassandra.db.SliceFromReadCommandSerializer.deserialize(SliceFromReadCommand.java:153)
> ~[apache-cassandra-2.1.2.jar:2.1.2]
> at
> org.apache.cassandra.db.ReadCommandSerializer.deserialize(ReadCommand.java:157)
> ~[apache-cassandra-2.1.2.jar:2.1.2]
> at
> org.apache.cassandra.db.ReadCommandSerializer.deserialize(ReadCommand.java:131)
> ~[apache-cassandra-2.1.2.jar:2.1.2]
> at org.apache.cassandra.net.MessageIn.read(MessageIn.java:99)
> ~[apache-cassandra-2.1.2.jar:2.1.2]
> at
> org.apache.cassandra.net.IncomingTcpConnection.receiveMessage(IncomingTcpConnection.java:168)
> ~[apache-cassandra-2.1.2.jar:2.1.2]
> at
> org.apache.cassandra.net.IncomingTcpConnection.receiveMessages(IncomingTcpConnection.java:150)
> ~[apache-cassandra-2.1.2.jar:2.1.2]
> at
> org.apache.cassandra.net.IncomingTcpConnection.run(IncomingTcpConnection.java:82)
> ~[apache-cassandra-2.1.2.jar:2.1.2]
> WARN [Thread-29065] 2015-01-15 16:10:56,374 IncomingTcpConnection.java:91 -
> UnknownColumnFamilyException reading from socket; closing
> org.apache.cassandra.db.UnknownColumnFamilyException: Couldn't find
> cfId=d1ea009c-feba-3d75-aa64-d022270842c4
> at
> org.apache.cassandra.db.ColumnFamilySerializer.deserializeCfId(ColumnFamilySerializer.java:164)
> ~[apache-cassandra-2.1.2.jar:2.1.2]
> at
> org.apache.cassandra.db.ColumnFamilySerializer.deserialize(ColumnFamilySerializer.java:97)
> ~[apache-cassandra-2.1.2.jar:2.1.2]
> at
> org.apache.cassandra.db.Mutation$MutationSerializer.deserializeOneCf(Mutation.java:322)
> ~[apache-cassandra-2.1.2.jar:2.1.2]
> at
> org.apache.cassandra.db.Mutation$MutationSerializer.deserialize(Mutation.java:302)
> ~[apache-cassandra-2.1.2.jar:2.1.2]
> at
> org.apache.cassandra.db.Mutation$MutationSerializer.deserialize(Mutation.java:330)
> ~[apache-cassandra-2.1.2.jar:2.1.2]
> at
> org.apache.cassandra.db.Mutation$MutationSerializer.deserialize(Mutation.java:272)
> ~[apache-cassandra-2.1.2.jar:2.1.2]
> at org.apache.cassandra.net.MessageIn.read(MessageIn.java:99)
> ~[apache-cassandra-2.1.2.jar:2.1.2]
> at
> org.apache.cassandra.net.IncomingTcpConnection.receiveMessage(IncomingTcpConnection.java:168)
> ~[apache-cassandra-2.1.2.jar:2.1.2]
> at
> org.apache.cassandra.net.IncomingTcpConnection.receiveMessages(IncomingTcpConnection.java:150)
> ~[apache-cassandra-2.1.2.jar:2.1.2]
> at
> org.apache.cassandra.net.IncomingTcpConnection.run(IncomingTcpConnection.java:82)
> ~[apache-cassandra-2.1.2.jar:2.1.2]
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
