Hi All, We followed the upgrade guide( http://www.datastax.com/docs/1.2/install/upgrading) from Datastax web site and upgraded Cassadra to 1.2.5, but it occurred errors in system.log when starting up.
After digging into code level, it looks like Cassandra found the file length of IndexSummary sstable is zero. Thus Cassandra threw AssertionError. In fact, the file length of the IndexSummary is about 80 bytes, not zero. It's weird. Also we observed that only happens on the IndexSummary file of secondary index. The errors can be reproducible. Below are my upgrade steps. 1. Shutdown all of client applications. 2. Run "nodetool drain" before shutting down the existing Cassandra service. 3. Stop old Cassandra process, then start the new binary process using migrated cassandra.yaml. 4. Run "nodetool upgradesstables -a" in order to upgrade all of sstable files become new format. 5. Restart Cassandra process and monitor the logs file for any issues. At step 5, we found the error messages as below. Any ideas? Thank you! Colin =================================================================================================================================== INFO [SSTableBatchOpen:2] 2013-05-29 04:38:40,085 SSTableReader.java (line 169) Opening /var/lib/cassandra/data/ks/user/ks-user.ks_user_personalID-ic-61 (58 bytes) ERROR [SSTableBatchOpen:1] 2013-05-29 04:38:40,085 CassandraDaemon.java (line 175) Exception in thread Thread[SSTableBatchOpen:1,5,main] java.lang.AssertionError at org.apache.cassandra.utils.ByteBufferUtil.readBytes(ByteBufferUtil.java:401) at org.apache.cassandra.io.sstable.IndexSummary$IndexSummarySerializer.deserialize(IndexSummary.java:124) at org.apache.cassandra.io.sstable.SSTableReader.loadSummary(SSTableReader.java:426) at org.apache.cassandra.io.sstable.SSTableReader.load(SSTableReader.java:360) at org.apache.cassandra.io.sstable.SSTableReader.open(SSTableReader.java:201) at org.apache.cassandra.io.sstable.SSTableReader.open(SSTableReader.java:154) at org.apache.cassandra.io.sstable.SSTableReader$1.run(SSTableReader.java:241) at java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source) at java.util.concurrent.FutureTask$Sync.innerRun(Unknown Source) at java.util.concurrent.FutureTask.run(Unknown Source) at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(Unknown Source) at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) at java.lang.Thread.run(Unknown Source) ERROR [SSTableBatchOpen:2] 2013-05-29 04:38:40,085 CassandraDaemon.java (line 175) Exception in thread Thread[SSTableBatchOpen:2,5,main] java.lang.AssertionError at org.apache.cassandra.utils.ByteBufferUtil.readBytes(ByteBufferUtil.java:401) at org.apache.cassandra.io.sstable.IndexSummary$IndexSummarySerializer.deserialize(IndexSummary.java:124) at org.apache.cassandra.io.sstable.SSTableReader.loadSummary(SSTableReader.java:426) at org.apache.cassandra.io.sstable.SSTableReader.load(SSTableReader.java:360) at org.apache.cassandra.io.sstable.SSTableReader.open(SSTableReader.java:201) at org.apache.cassandra.io.sstable.SSTableReader.open(SSTableReader.java:154) at org.apache.cassandra.io.sstable.SSTableReader$1.run(SSTableReader.java:241) at java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source) at java.util.concurrent.FutureTask$Sync.innerRun(Unknown Source) at java.util.concurrent.FutureTask.run(Unknown Source) at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(Unknown Source) at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) at java.lang.Thread.run(Unknown Source) ===================================================================================================================================
