[jira] [Updated] (CASSANDRA-6493) Exceptions when a second Datacenter is Added
[
https://issues.apache.org/jira/browse/CASSANDRA-6493?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Russell Alexander Spitzer updated CASSANDRA-6493:
-
Description:
On adding a second datacenter several exceptions were raised.
Test outline:
Start 25 Node DC1
Keyspace Setup Replication 3
Begin insert against DC1 Using Stress
While the inserts are occuring
Start up 25 Node DC2
Alter Keyspace to include Replication in 2nd DC
Run rebuild on DC2
Wait for stress to finish
Run repair on Cluster
... Some other operations
At the point when the second datacenter is added several warnings go off
because nodetool status is not functioning, and a few moments later the start
operation reports a failure because a node has not successfully turned on.
The first start attempt yielded the following exception on a node in the second
DC.
{code}
CassandraDaemon.java (line 464) Exception encountered during startup
java.lang.AssertionError: -7560216458456714666 not found in
-9222060278673125462, -9220751250790085193, . ALL THE TOKENS ...,
9218575851928340117, 9219681798686280387
at
org.apache.cassandra.locator.TokenMetadata.getPredecessor(TokenMetadata.java:752)
at
org.apache.cassandra.locator.TokenMetadata.getPrimaryRangesFor(TokenMetadata.java:696)
at
org.apache.cassandra.locator.TokenMetadata.getPrimaryRangeFor(TokenMetadata.java:703)
at
org.apache.cassandra.locator.AbstractReplicationStrategy.getRangeAddresses(AbstractReplicationStrategy.java:187)
at
org.apache.cassandra.dht.RangeStreamer.getAllRangesWithSourcesFor(RangeStreamer.java:147)
at org.apache.cassandra.dht.RangeStreamer.addRanges(RangeStreamer.java:121)
at org.apache.cassandra.dht.BootStrapper.bootstrap(BootStrapper.java:81)
at
org.apache.cassandra.service.StorageService.bootstrap(StorageService.java:979)
at
org.apache.cassandra.service.StorageService.joinTokenRing(StorageService.java:745)
at
org.apache.cassandra.service.StorageService.initServer(StorageService.java:586)
at
org.apache.cassandra.service.StorageService.initServer(StorageService.java:483)
at
org.apache.cassandra.service.CassandraDaemon.setup(CassandraDaemon.java:348)
at
org.apache.cassandra.service.CassandraDaemon.activate(CassandraDaemon.java:447)
at
org.apache.cassandra.service.CassandraDaemon.main(CassandraDaemon.java:490)
{code}
The test automatically tries to restart nodes if they fail during startup, The
second attempt for this node succeeded but a 'nodetool status' still failed and
a different node in the second DC logged the following and failed to start up.
{code}
ERROR [main] 2013-12-16 18:02:04,869 CassandraDaemon.java (line 464) Exception
encountered during startup
java.util.ConcurrentModificationException
at java.util.TreeMap$PrivateEntryIterator.nextEntry(TreeMap.java:1115)
at java.util.TreeMap$KeyIterator.next(TreeMap.java:1169)
at org.apache.commons.lang.StringUtils.join(StringUtils.java:3382)
at org.apache.commons.lang.StringUtils.join(StringUtils.java:3444)
at
org.apache.cassandra.locator.TokenMetadata.getPredecessor(TokenMetadata.java:752)
at
org.apache.cassandra.locator.TokenMetadata.getPrimaryRangesFor(TokenMetadata.java:696)
at
org.apache.cassandra.locator.TokenMetadata.getPrimaryRangeFor(TokenMetadata.java:703)
at
org.apache.cassandra.locator.AbstractReplicationStrategy.getRangeAddresses(AbstractReplicationStrategy.java:187)
at
org.apache.cassandra.dht.RangeStreamer.getAllRangesWithSourcesFor(RangeStreamer.java:147)
at
org.apache.cassandra.dht.RangeStreamer.addRanges(RangeStreamer.java:121)
at org.apache.cassandra.dht.BootStrapper.bootstrap(BootStrapper.java:81)
at
org.apache.cassandra.service.StorageService.bootstrap(StorageService.java:979)
at
org.apache.cassandra.service.StorageService.joinTokenRing(StorageService.java:745)
at
org.apache.cassandra.service.StorageService.initServer(StorageService.java:586)
at
org.apache.cassandra.service.StorageService.initServer(StorageService.java:483)
at
org.apache.cassandra.service.CassandraDaemon.setup(CassandraDaemon.java:348)
at
org.apache.cassandra.service.CassandraDaemon.activate(CassandraDaemon.java:447)
at
org.apache.cassandra.service.CassandraDaemon.main(CassandraDaemon.java:490)
ERROR [StorageServiceShutdownHook] 2013-12-16 18:02:04,876 CassandraDaemon.java
(line 191) Exception in thread Thread[StorageServiceShutdownHook,5,main]
java.lang.NullPointerException
at
org.apache.cassandra.service.StorageService.stopNativeTransport(StorageService.java:358)
at
org.apache.cassandra.service.StorageService.shutdownClientServers(StorageService.java:373)
at
org.apache.cassandra.service.StorageService.access$000(StorageService.java:89)
at
[jira] [Updated] (CASSANDRA-6493) Exceptions when a second Datacenter is Added
[
https://issues.apache.org/jira/browse/CASSANDRA-6493?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Russell Alexander Spitzer updated CASSANDRA-6493:
-
Reproduced In: 1.2.13
Exceptions when a second Datacenter is Added
Key: CASSANDRA-6493
URL: https://issues.apache.org/jira/browse/CASSANDRA-6493
Project: Cassandra
Issue Type: Bug
Components: Core
Environment: Ubuntu, EC2 M1.large
Reporter: Russell Alexander Spitzer
On adding a second datacenter several exceptions were raised.
Test outline:
Start 25 Node DC1
Keyspace Setup Replication 3
Begin insert against DC1 Using Stress
While the inserts are occuring
Start up 25 Node DC2
Alter Keyspace to include Replication in 2nd DC
Run rebuild on DC2
Wait for stress to finish
Run repair on Cluster
... Some other operations
At the point when the second datacenter is added several warnings go off
because nodetool status is not functioning, and a few moments later the start
operation reports a failure because a node has not successfully turned on.
The first start attempt yielded the following exception on a node in the
second DC.
{code}
CassandraDaemon.java (line 464) Exception encountered during startup
java.lang.AssertionError: -7560216458456714666 not found in
-9222060278673125462, -9220751250790085193, . ALL THE TOKENS ...,
9218575851928340117, 9219681798686280387
at
org.apache.cassandra.locator.TokenMetadata.getPredecessor(TokenMetadata.java:752)
at
org.apache.cassandra.locator.TokenMetadata.getPrimaryRangesFor(TokenMetadata.java:696)
at
org.apache.cassandra.locator.TokenMetadata.getPrimaryRangeFor(TokenMetadata.java:703)
at
org.apache.cassandra.locator.AbstractReplicationStrategy.getRangeAddresses(AbstractReplicationStrategy.java:187)
at
org.apache.cassandra.dht.RangeStreamer.getAllRangesWithSourcesFor(RangeStreamer.java:147)
at org.apache.cassandra.dht.RangeStreamer.addRanges(RangeStreamer.java:121)
at org.apache.cassandra.dht.BootStrapper.bootstrap(BootStrapper.java:81)
at
org.apache.cassandra.service.StorageService.bootstrap(StorageService.java:979)
at
org.apache.cassandra.service.StorageService.joinTokenRing(StorageService.java:745)
at
org.apache.cassandra.service.StorageService.initServer(StorageService.java:586)
at
org.apache.cassandra.service.StorageService.initServer(StorageService.java:483)
at
org.apache.cassandra.service.CassandraDaemon.setup(CassandraDaemon.java:348)
at
org.apache.cassandra.service.CassandraDaemon.activate(CassandraDaemon.java:447)
at
org.apache.cassandra.service.CassandraDaemon.main(CassandraDaemon.java:490)
{code}
The test automatically tries to restart nodes if they fail during startup,
The second attempt for this node succeeded but a nodetool still failed and a
different node in the second DC logged the following and failed to start up.
{code}
ERROR [main] 2013-12-16 18:02:04,869 CassandraDaemon.java (line 464)
Exception encountered during startup
java.util.ConcurrentModificationException
at java.util.TreeMap$PrivateEntryIterator.nextEntry(TreeMap.java:1115)
at java.util.TreeMap$KeyIterator.next(TreeMap.java:1169)
at org.apache.commons.lang.StringUtils.join(StringUtils.java:3382)
at org.apache.commons.lang.StringUtils.join(StringUtils.java:3444)
at
org.apache.cassandra.locator.TokenMetadata.getPredecessor(TokenMetadata.java:752)
at
org.apache.cassandra.locator.TokenMetadata.getPrimaryRangesFor(TokenMetadata.java:696)
at
org.apache.cassandra.locator.TokenMetadata.getPrimaryRangeFor(TokenMetadata.java:703)
at
org.apache.cassandra.locator.AbstractReplicationStrategy.getRangeAddresses(AbstractReplicationStrategy.java:187)
at
org.apache.cassandra.dht.RangeStreamer.getAllRangesWithSourcesFor(RangeStreamer.java:147)
at
org.apache.cassandra.dht.RangeStreamer.addRanges(RangeStreamer.java:121)
at org.apache.cassandra.dht.BootStrapper.bootstrap(BootStrapper.java:81)
at
org.apache.cassandra.service.StorageService.bootstrap(StorageService.java:979)
at
org.apache.cassandra.service.StorageService.joinTokenRing(StorageService.java:745)
at
org.apache.cassandra.service.StorageService.initServer(StorageService.java:586)
at
org.apache.cassandra.service.StorageService.initServer(StorageService.java:483)
at
org.apache.cassandra.service.CassandraDaemon.setup(CassandraDaemon.java:348)
at
org.apache.cassandra.service.CassandraDaemon.activate(CassandraDaemon.java:447)
at
org.apache.cassandra.service.CassandraDaemon.main(CassandraDaemon.java:490)
ERROR [StorageServiceShutdownHook] 2013-12-16 18:02:04,876
CassandraDaemon.java (line 191) Exception in thread
