[
https://issues.apache.org/jira/browse/CASSANDRA-8628?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Lennart Liberg updated CASSANDRA-8628:
--------------------------------------
Summary: Stalling rebuild of new 2.1.2 node from cluster running 2.0.9
(was: Stalling rebuild of new 2.1.12 node from cluster running 2.0.9)
> Stalling rebuild of new 2.1.2 node from cluster running 2.0.9
> -------------------------------------------------------------
>
> Key: CASSANDRA-8628
> URL: https://issues.apache.org/jira/browse/CASSANDRA-8628
> Project: Cassandra
> Issue Type: Bug
> Components: Core
> Environment: Ubuntu 14.04, 4 core machines, 8 and 16 GB RAM
> Cluster running cassandra 2.0.9, new node running 2.1.2,
> inter-node encryption: all
> Reporter: Lennart Liberg
>
> We want to add a new datacenter to our existing cluster consisting of two
> datacenters containing 2 nodes and 1 node respectively, running version 2.0.9
> We configured a node running cassandra 2.1.2 in the new datacenter
> (auto_bootstrap: false) and verified that it managed to join the cluster as
> expected (`nodetool status` saying its status was UN).
> Upon executing `nodetool rebuild ams1`, the system.log output below is
> observed, with a new UnknownColumnFamilyException and NullPointerException
> every few minutes, and no data being transfered.
> After downgrading to cassandra 2.0.11, the rebuild operation worked flawless,
> using a cassandra.yaml configured as identical as possible to the 2.1.2 yaml
> file (a lot of new items in 2.1.x).
> My reading of the upgrading compatibility document leads me to think that
> there should not be any problem in performing this kind of rebuild.
> http://www.datastax.com/documentation/upgrade/doc/upgrade/cassandra/upgradeDebC_t.html
> system.log output:
> {noformat}
> INFO [RMI TCP Connection(57622)-10.240.12.239] 2015-01-15 16:00:04,228
> StorageService.java:958 - rebuild from dc: ams1
> INFO [RMI TCP Connection(57622)-10.240.12.239] 2015-01-15 16:00:04,305
> StreamResultFuture.java:86 - [Stream #91906440-9ccf-11e4-9cab-67625edb9342]
> Executing streaming plan for Rebuild
> INFO [StreamConnectionEstablisher:4] 2015-01-15 16:00:04,306
> StreamSession.java:213 - [Stream #91906440-9ccf-11e4-9cab-67625edb9342]
> Starting streaming to 37.xx.xx.1/37.xx.xx.1
> INFO [StreamConnectionEstablisher:4] 2015-01-15 16:00:04,610
> StreamCoordinator.java:209 - [Stream #91906440-9ccf-11e4-9cab-67625edb9342,
> ID#0] Beginning stream session with 37.xx.xx.1/37.xx.xx.1
> WARN [Thread-28935] 2015-01-15 16:03:40,458 IncomingTcpConnection.java:91 -
> UnknownColumnFamilyException reading from socket; closing
> org.apache.cassandra.db.UnknownColumnFamilyException: Couldn't find
> cfId=d1ea009c-feba-3d75-aa64-d022270842c4
> at
> org.apache.cassandra.db.ColumnFamilySerializer.deserializeCfId(ColumnFamilySerializer.java:164)
> ~[apache-cassandra-2.1.2.jar:2.1.2]
> at
> org.apache.cassandra.db.ColumnFamilySerializer.deserialize(ColumnFamilySerializer.java:97)
> ~[apache-cassandra-2.1.2.jar:2.1.2]
> at
> org.apache.cassandra.db.Mutation$MutationSerializer.deserializeOneCf(Mutation.java:322)
> ~[apache-cassandra-2.1.2.jar:2.1.2]
> at
> org.apache.cassandra.db.Mutation$MutationSerializer.deserialize(Mutation.java:302)
> ~[apache-cassandra-2.1.2.jar:2.1.2]
> at
> org.apache.cassandra.db.Mutation$MutationSerializer.deserialize(Mutation.java:330)
> ~[apache-cassandra-2.1.2.jar:2.1.2]
> at
> org.apache.cassandra.db.Mutation$MutationSerializer.deserialize(Mutation.java:272)
> ~[apache-cassandra-2.1.2.jar:2.1.2]
> at org.apache.cassandra.net.MessageIn.read(MessageIn.java:99)
> ~[apache-cassandra-2.1.2.jar:2.1.2]
> at
> org.apache.cassandra.net.IncomingTcpConnection.receiveMessage(IncomingTcpConnection.java:168)
> ~[apache-cassandra-2.1.2.jar:2.1.2]
> at
> org.apache.cassandra.net.IncomingTcpConnection.receiveMessages(IncomingTcpConnection.java:150)
> ~[apache-cassandra-2.1.2.jar:2.1.2]
> at
> org.apache.cassandra.net.IncomingTcpConnection.run(IncomingTcpConnection.java:82)
> ~[apache-cassandra-2.1.2.jar:2.1.2]
> ERROR [Thread-28974] 2015-01-15 16:05:54,164 CassandraDaemon.java:153 -
> Exception in thread Thread[Thread-28974,5,main]
> java.lang.NullPointerException: null
> at
> org.apache.cassandra.db.SliceFromReadCommandSerializer.deserialize(SliceFromReadCommand.java:153)
> ~[apache-cassandra-2.1.2.jar:2.1.2]
> at
> org.apache.cassandra.db.ReadCommandSerializer.deserialize(ReadCommand.java:157)
> ~[apache-cassandra-2.1.2.jar:2.1.2]
> at
> org.apache.cassandra.db.ReadCommandSerializer.deserialize(ReadCommand.java:131)
> ~[apache-cassandra-2.1.2.jar:2.1.2]
> at org.apache.cassandra.net.MessageIn.read(MessageIn.java:99)
> ~[apache-cassandra-2.1.2.jar:2.1.2]
> at
> org.apache.cassandra.net.IncomingTcpConnection.receiveMessage(IncomingTcpConnection.java:168)
> ~[apache-cassandra-2.1.2.jar:2.1.2]
> at
> org.apache.cassandra.net.IncomingTcpConnection.receiveMessages(IncomingTcpConnection.java:150)
> ~[apache-cassandra-2.1.2.jar:2.1.2]
> at
> org.apache.cassandra.net.IncomingTcpConnection.run(IncomingTcpConnection.java:82)
> ~[apache-cassandra-2.1.2.jar:2.1.2]
> ERROR [Thread-29011] 2015-01-15 16:07:58,349 CassandraDaemon.java:153 -
> Exception in thread Thread[Thread-29011,5,main]
> java.lang.NullPointerException: null
> at
> org.apache.cassandra.db.SliceFromReadCommandSerializer.deserialize(SliceFromReadCommand.java:153)
> ~[apache-cassandra-2.1.2.jar:2.1.2]
> at
> org.apache.cassandra.db.ReadCommandSerializer.deserialize(ReadCommand.java:157)
> ~[apache-cassandra-2.1.2.jar:2.1.2]
> at
> org.apache.cassandra.db.ReadCommandSerializer.deserialize(ReadCommand.java:131)
> ~[apache-cassandra-2.1.2.jar:2.1.2]
> at org.apache.cassandra.net.MessageIn.read(MessageIn.java:99)
> ~[apache-cassandra-2.1.2.jar:2.1.2]
> at
> org.apache.cassandra.net.IncomingTcpConnection.receiveMessage(IncomingTcpConnection.java:168)
> ~[apache-cassandra-2.1.2.jar:2.1.2]
> at
> org.apache.cassandra.net.IncomingTcpConnection.receiveMessages(IncomingTcpConnection.java:150)
> ~[apache-cassandra-2.1.2.jar:2.1.2]
> at
> org.apache.cassandra.net.IncomingTcpConnection.run(IncomingTcpConnection.java:82)
> ~[apache-cassandra-2.1.2.jar:2.1.2]
> ERROR [Thread-29013] 2015-01-15 16:08:01,037 CassandraDaemon.java:153 -
> Exception in thread Thread[Thread-29013,5,main]
> java.lang.NullPointerException: null
> at
> org.apache.cassandra.db.SliceFromReadCommandSerializer.deserialize(SliceFromReadCommand.java:153)
> ~[apache-cassandra-2.1.2.jar:2.1.2]
> at
> org.apache.cassandra.db.ReadCommandSerializer.deserialize(ReadCommand.java:157)
> ~[apache-cassandra-2.1.2.jar:2.1.2]
> at
> org.apache.cassandra.db.ReadCommandSerializer.deserialize(ReadCommand.java:131)
> ~[apache-cassandra-2.1.2.jar:2.1.2]
> at org.apache.cassandra.net.MessageIn.read(MessageIn.java:99)
> ~[apache-cassandra-2.1.2.jar:2.1.2]
> at
> org.apache.cassandra.net.IncomingTcpConnection.receiveMessage(IncomingTcpConnection.java:168)
> ~[apache-cassandra-2.1.2.jar:2.1.2]
> at
> org.apache.cassandra.net.IncomingTcpConnection.receiveMessages(IncomingTcpConnection.java:150)
> ~[apache-cassandra-2.1.2.jar:2.1.2]
> at
> org.apache.cassandra.net.IncomingTcpConnection.run(IncomingTcpConnection.java:82)
> ~[apache-cassandra-2.1.2.jar:2.1.2]
> WARN [Thread-29065] 2015-01-15 16:10:56,374 IncomingTcpConnection.java:91 -
> UnknownColumnFamilyException reading from socket; closing
> org.apache.cassandra.db.UnknownColumnFamilyException: Couldn't find
> cfId=d1ea009c-feba-3d75-aa64-d022270842c4
> at
> org.apache.cassandra.db.ColumnFamilySerializer.deserializeCfId(ColumnFamilySerializer.java:164)
> ~[apache-cassandra-2.1.2.jar:2.1.2]
> at
> org.apache.cassandra.db.ColumnFamilySerializer.deserialize(ColumnFamilySerializer.java:97)
> ~[apache-cassandra-2.1.2.jar:2.1.2]
> at
> org.apache.cassandra.db.Mutation$MutationSerializer.deserializeOneCf(Mutation.java:322)
> ~[apache-cassandra-2.1.2.jar:2.1.2]
> at
> org.apache.cassandra.db.Mutation$MutationSerializer.deserialize(Mutation.java:302)
> ~[apache-cassandra-2.1.2.jar:2.1.2]
> at
> org.apache.cassandra.db.Mutation$MutationSerializer.deserialize(Mutation.java:330)
> ~[apache-cassandra-2.1.2.jar:2.1.2]
> at
> org.apache.cassandra.db.Mutation$MutationSerializer.deserialize(Mutation.java:272)
> ~[apache-cassandra-2.1.2.jar:2.1.2]
> at org.apache.cassandra.net.MessageIn.read(MessageIn.java:99)
> ~[apache-cassandra-2.1.2.jar:2.1.2]
> at
> org.apache.cassandra.net.IncomingTcpConnection.receiveMessage(IncomingTcpConnection.java:168)
> ~[apache-cassandra-2.1.2.jar:2.1.2]
> at
> org.apache.cassandra.net.IncomingTcpConnection.receiveMessages(IncomingTcpConnection.java:150)
> ~[apache-cassandra-2.1.2.jar:2.1.2]
> at
> org.apache.cassandra.net.IncomingTcpConnection.run(IncomingTcpConnection.java:82)
> ~[apache-cassandra-2.1.2.jar:2.1.2]
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
