Brooke Bryan created CASSANDRA-5370:
---------------------------------------
Summary: Restart Corrupting system.local
Key: CASSANDRA-5370
URL: https://issues.apache.org/jira/browse/CASSANDRA-5370
Project: Cassandra
Issue Type: Bug
Affects Versions: 1.2.2
Environment: 1.2.2
Reporter: Brooke Bryan
Priority: Critical
Im not too sure on the best way to explain this, however, when we have been
upgrading nodes, from 1.1.9 and 1.2.0 up to v 1.2.2, we have found some nodes
are corrupting on their second or third boot / restart.
The nodes would be happily sat in the ring, running normally, and we would
upgrade them to 1.2.2, and they would continue to run fine. However, on a
restart, the cluster would then think its part of a Test Cluster. We did see a
few nodes that were running in the main cluster, but "SELECT cluster_name FROM
system.local;" would return Test Cluster. Updating this, and restarting would
fix the issue, however, I have no idea how the cluster name would have changed
after boot.
INFO [main] 2013-03-21 05:14:01,561 CassandraDaemon.java (line 100) Logging
initialized
INFO [main] 2013-03-21 05:14:01,581 CassandraDaemon.java (line 122) JVM
vendor/version: Java HotSpot(TM) 64-Bit Server VM/1.7.0
INFO [main] 2013-03-21 05:14:01,581 CassandraDaemon.java (line 123) Heap size:
10653532160/10653532160
INFO [main] 2013-03-21 05:14:01,581 CassandraDaemon.java (line 124) Classpath:
/usr/local/share/cassandra/bin/../conf:/usr/local/share/cassandra/bin/../build/classes/main:/usr/local/share/cassandra/bin/../build/classes/thrift:/usr/local/share/cassandra/bin/../lib/antlr-3.2.jar:/usr/local/share/cassandra/bin/../lib/apache-cassandra-1.2.2.jar:/usr/local/share/cassandra/bin/../lib/apache-cassandra-clientutil-1.2.2.jar:/usr/local/share/cassandra/bin/../lib/apache-cassandra-thrift-1.2.2.jar:/usr/local/share/cassandra/bin/../lib/avro-1.4.0-fixes.jar:/usr/local/share/cassandra/bin/../lib/avro-1.4.0-sources-fixes.jar:/usr/local/share/cassandra/bin/../lib/commons-cli-1.1.jar:/usr/local/share/cassandra/bin/../lib/commons-codec-1.2.jar:/usr/local/share/cassandra/bin/../lib/commons-lang-2.6.jar:/usr/local/share/cassandra/bin/../lib/compress-lzf-0.8.4.jar:/usr/local/share/cassandra/bin/../lib/concurrentlinkedhashmap-lru-1.3.jar:/usr/local/share/cassandra/bin/../lib/guava-13.0.1.jar:/usr/local/share/cassandra/bin/../lib/high-scale-lib-1.1.2.jar:/usr/local/share/cassandra/bin/../lib/jackson-core-asl-1.9.2.jar:/usr/local/share/cassandra/bin/../lib/jackson-mapper-asl-1.9.2.jar:/usr/local/share/cassandra/bin/../lib/jamm-0.2.5.jar:/usr/local/share/cassandra/bin/../lib/jbcrypt-0.3m.jar:/usr/local/share/cassandra/bin/../lib/jline-1.0.jar:/usr/local/share/cassandra/bin/../lib/jna.jar:/usr/local/share/cassandra/bin/../lib/json-simple-1.1.jar:/usr/local/share/cassandra/bin/../lib/libthrift-0.7.0.jar:/usr/local/share/cassandra/bin/../lib/log4j-1.2.16.jar:/usr/local/share/cassandra/bin/../lib/lz4-1.1.0.jar:/usr/local/share/cassandra/bin/../lib/metrics-core-2.0.3.jar:/usr/local/share/cassandra/bin/../lib/mx4j-tools.jar:/usr/local/share/cassandra/bin/../lib/netty-3.5.9.Final.jar:/usr/local/share/cassandra/bin/../lib/servlet-api-2.5-20081211.jar:/usr/local/share/cassandra/bin/../lib/slf4j-api-1.7.2.jar:/usr/local/share/cassandra/bin/../lib/slf4j-log4j12-1.7.2.jar:/usr/local/share/cassandra/bin/../lib/snakeyaml-1.6.jar:/usr/local/share/cassandra/bin/../lib/snappy-java-1.0.4.1.jar:/usr/local/share/cassandra/bin/../lib/snaptree-0.1.jar:/usr/local/share/cassandra/bin/../lib/jamm-0.2.5.jar
INFO [main] 2013-03-21 05:14:01,601 CLibrary.java (line 65) JNA link failure,
one or more native method will be unavailable.
INFO [main] 2013-03-21 05:14:01,612 DatabaseDescriptor.java (line 131) Loading
settings from file:/usr/local/share/cassandra/conf/cassandra.yaml
INFO [main] 2013-03-21 05:14:01,908 DatabaseDescriptor.java (line 190)
DiskAccessMode 'auto' determined to be mmap, indexAccessMode is mmap
INFO [main] 2013-03-21 05:14:01,908 DatabaseDescriptor.java (line 204)
disk_failure_policy is stop
INFO [main] 2013-03-21 05:14:01,913 DatabaseDescriptor.java (line 265) Global
memtable threshold is enabled at 3386MB
INFO [main] 2013-03-21 05:14:02,438 CacheService.java (line 111) Initializing
key cache with capacity of 100 MBs.
INFO [main] 2013-03-21 05:14:02,446 CacheService.java (line 140) Scheduling
key cache save to each 14400 seconds (going to save all keys).
INFO [main] 2013-03-21 05:14:02,447 CacheService.java (line 154) Initializing
row cache with capacity of 0 MBs and provider
org.apache.cassandra.cache.SerializingCacheProvider
INFO [main] 2013-03-21 05:14:02,452 CacheService.java (line 166) Scheduling
row cache save to each 0 seconds (going to save all keys).
--
INFO [SSTableBatchOpen:1] 2013-03-21 05:14:02,667 SSTableReader.java (line
164) Opening /cassandra_data/system/peers/system-peers-ib-42 (525 bytes)
INFO [SSTableBatchOpen:2] 2013-03-21 05:14:02,668 SSTableReader.java (line
164) Opening /cassandra_data/system/peers/system-peers-ib-41 (25308 bytes)
INFO [SSTableBatchOpen:1] 2013-03-21 05:14:02,678 SSTableReader.java (line
164) Opening /cassandra_data/system/local/system-local-ib-48 (84 bytes)
INFO [SSTableBatchOpen:2] 2013-03-21 05:14:02,678 SSTableReader.java (line
164) Opening /cassandra_data/system/local/system-local-ib-47 (84 bytes)
INFO [SSTableBatchOpen:3] 2013-03-21 05:14:02,679 SSTableReader.java (line
164) Opening /cassandra_data/system/local/system-local-ib-46 (3151 bytes)
ERROR [main] 2013-03-21 05:14:02,780 CassandraDaemon.java (line 180) Fatal
exception during initialization
org.apache.cassandra.exceptions.ConfigurationException: Saved cluster name Test
Cluster != configured name Meta Cluster
at org.apache.cassandra.db.SystemTable.checkHealth(SystemTable.java:452)
at
org.apache.cassandra.service.CassandraDaemon.setup(CassandraDaemon.java:176)
at
org.apache.cassandra.service.CassandraDaemon.activate(CassandraDaemon.java:366)
at
org.apache.cassandra.service.CassandraDaemon.main(CassandraDaemon.java:409)
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
