[
https://issues.apache.org/jira/browse/CASSANDRA-1365?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jonathan Ellis resolved CASSANDRA-1365.
---------------------------------------
Resolution: Invalid
if you're seeing NoSuchMethodError, it means you need to run "ant clean". (if
it's extra stubborn, use "ant realclean")
> Error during compaction followed by error during commitlog replay
> -----------------------------------------------------------------
>
> Key: CASSANDRA-1365
> URL: https://issues.apache.org/jira/browse/CASSANDRA-1365
> Project: Cassandra
> Issue Type: Bug
> Components: Core
> Affects Versions: 0.7.0
> Environment: OSX 10.6.4
> java version "1.6.0_20"
> Java(TM) SE Runtime Environment (build 1.6.0_20-b02-279-10M3065)
> Java HotSpot(TM) 64-Bit Server VM (build 16.3-b01-279, mixed mode)
> cassandra r982829
> Reporter: Valentino Volonghi
>
> Created a cassandra ring of 2 nodes. Node 1 seeds node 2 and node 2 seeds
> node 1.
> 1) Fresh nodes up and verify that they both saw each other.
> 2) Run: python stress.py -o insert -n 100000 -d 192.168.101.168 -y regular -c
> 20 -i 5
> 3) Run: ./bin/nodetool -h 192.168.101.168 flush Keyspace1
> 3) Run: ./bin/nodetool -h 192.168.101.168 snapshot
> => checking the console output of cassandra this is the logging I see:
> {noformat}
> INFO 18:27:06,597 switching in a fresh Memtable for LocationInfo at
> CommitLogContext(file='/var/lib/cassandra/commitlog/CommitLog-1281057918124.log',
> position=95597256)
> INFO 18:27:06,598 Enqueuing flush of memtable-locationi...@1029661208(52
> bytes, 2 operations)
> INFO 18:27:06,599 Writing memtable-locationi...@1029661208(52 bytes, 2
> operations)
> INFO 18:27:06,704 Completed flushing
> /var/lib/cassandra/data/system/LocationInfo-e-4-Data.db
> DEBUG 18:27:06,705 key cache capacity for LocationInfo is 5
> DEBUG 18:27:06,705 Checking to see if compaction of LocationInfo would be
> useful
> INFO 18:27:06,706 Compacting
> [org.apache.cassandra.io.sstable.SSTableReader(path='/var/lib/cassandra/data/system/LocationInfo-e-1-Data.db'),org.apache.cassandra.io.sstable.SSTableReader(path='/var/lib/cassandra/data/system/LocationInfo-e-2-Data.db'),org.apache.cassandra.io.sstable.SSTableReader(path='/var/lib/cassandra/data/system/LocationInfo-e-3-Data.db'),org.apache.cassandra.io.sstable.SSTableReader(path='/var/lib/cassandra/data/system/LocationInfo-e-4-Data.db')]
> DEBUG 18:27:06,706 Discarding 0
> DEBUG 18:27:06,706 discard completed log segments for
> CommitLogContext(file='/var/lib/cassandra/commitlog/CommitLog-1281057918124.log',
> position=95597256), column family 0.
> DEBUG 18:27:06,706 index size for bloom filter calc for file :
> /var/lib/cassandra/data/system/LocationInfo-e-1-Data.db : 256
> DEBUG 18:27:06,708 Not safe to delete commit log
> CommitLogSegment(/var/lib/cassandra/commitlog/CommitLog-1281057874925.log);
> dirty is 2, 3,
> DEBUG 18:27:06,709 index size for bloom filter calc for file :
> /var/lib/cassandra/data/system/LocationInfo-e-2-Data.db : 512
> DEBUG 18:27:06,709 index size for bloom filter calc for file :
> /var/lib/cassandra/data/system/LocationInfo-e-3-Data.db : 768
> DEBUG 18:27:06,709 index size for bloom filter calc for file :
> /var/lib/cassandra/data/system/LocationInfo-e-4-Data.db : 1024
> DEBUG 18:27:06,709 Expected bloom filter size : 1024
> DEBUG 18:27:06,709 Marking replay position 95597256 on commit log
> CommitLogSegment(/var/lib/cassandra/commitlog/CommitLog-1281057918124.log)
> ERROR 18:27:06,752 Uncaught exception in thread
> Thread[CompactionExecutor:1,5,main]
> java.util.concurrent.ExecutionException: java.lang.NoSuchMethodError:
> org.apache.cassandra.config.DatabaseDescriptor.getInMemoryCompactionLimit()J
> at java.util.concurrent.FutureTask$Sync.innerGet(FutureTask.java:222)
> at java.util.concurrent.FutureTask.get(FutureTask.java:83)
> at
> org.apache.cassandra.concurrent.DebuggableThreadPoolExecutor.afterExecute(DebuggableThreadPoolExecutor.java:87)
> at
> org.apache.cassandra.db.CompactionManager$CompactionExecutor.afterExecute(CompactionManager.java:650)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:888)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> at java.lang.Thread.run(Thread.java:637)
> Caused by: java.lang.NoSuchMethodError:
> org.apache.cassandra.config.DatabaseDescriptor.getInMemoryCompactionLimit()J
> at
> org.apache.cassandra.io.CompactionIterator.getCompactedRow(CompactionIterator.java:132)
> at
> org.apache.cassandra.io.CompactionIterator.getReduced(CompactionIterator.java:107)
> at
> org.apache.cassandra.io.CompactionIterator.getReduced(CompactionIterator.java:46)
> at
> org.apache.cassandra.utils.ReducingIterator.computeNext(ReducingIterator.java:73)
> at
> com.google.common.collect.AbstractIterator.tryToComputeNext(AbstractIterator.java:136)
> at
> com.google.common.collect.AbstractIterator.hasNext(AbstractIterator.java:131)
> at
> org.apache.commons.collections.iterators.FilterIterator.setNextObject(FilterIterator.java:183)
> at
> org.apache.commons.collections.iterators.FilterIterator.hasNext(FilterIterator.java:94)
> at
> org.apache.cassandra.db.CompactionManager.doCompaction(CompactionManager.java:333)
> at
> org.apache.cassandra.db.CompactionManager$2.call(CompactionManager.java:151)
> at
> org.apache.cassandra.db.CompactionManager$2.call(CompactionManager.java:132)
> at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
> at java.util.concurrent.FutureTask.run(FutureTask.java:138)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> ... 2 more
> DEBUG 18:27:06,846 Snapshot for system table data file
> /var/lib/cassandra/data/system/LocationInfo-e-3-Filter.db created as
> /var/lib/cassandra/data/system/snapshots/1281058026597/LocationInfo-e-3-Filter.db
> DEBUG 18:27:06,976 Snapshot for system table data file
> /var/lib/cassandra/data/system/LocationInfo-e-1-Filter.db created as
> /var/lib/cassandra/data/system/snapshots/1281058026597/LocationInfo-e-1-Filter.db
> DEBUG 18:27:07,091 Snapshot for system table data file
> /var/lib/cassandra/data/system/LocationInfo-e-2-Filter.db created as
> /var/lib/cassandra/data/system/snapshots/1281058026597/LocationInfo-e-2-Filter.db
> DEBUG 18:27:07,210 Snapshot for system table data file
> /var/lib/cassandra/data/system/LocationInfo-e-4-Filter.db created as
> /var/lib/cassandra/data/system/snapshots/1281058026597/LocationInfo-e-4-Filter.db
> DEBUG 18:27:07,341 Snapshot for system table data file
> /var/lib/cassandra/data/system/Migrations-e-1-Filter.db created as
> /var/lib/cassandra/data/system/snapshots/1281058026597/Migrations-e-1-Filter.db
> DEBUG 18:27:07,462 Snapshot for system table data file
> /var/lib/cassandra/data/system/Schema-e-1-Filter.db created as
> /var/lib/cassandra/data/system/snapshots/1281058026597/Schema-e-1-Filter.db
> {noformat}
> 4) At this point simply stop and restart cassandra. The first restart would
> present this logging, the subsequent ones similar logging:
> {noformat}
> INFO 18:32:08,747 Replaying
> /var/lib/cassandra/commitlog/CommitLog-1281057874925.log,
> /var/lib/cassandra/commitlog/CommitLog-1281057918124.log
> DEBUG 18:32:08,749 Replaying
> /var/lib/cassandra/commitlog/CommitLog-1281057874925.log starting at 8974
> INFO 18:32:08,749 Finished reading
> /var/lib/cassandra/commitlog/CommitLog-1281057874925.log
> DEBUG 18:32:08,790 Replaying
> /var/lib/cassandra/commitlog/CommitLog-1281057918124.log starting at 95589044
> DEBUG 18:32:08,791 Reading mutation at 95589044
> DEBUG 18:32:08,800 replaying mutation for system...@60c0c8b5:
> {ColumnFamily(Statistics [SuperColumn(ColumnCount
> [0000000000000001:false:8...@1281058015446,0000000000000002:false:8...@1281058015446,0000000000000003:false:8...@1281058015446,0000000000000004:false:8...@1281058015446,0000000000000005:false:8...@1281058015446,0000000000000006:false:8...@1281058015446,0000000000000007:false:8...@1281058015446,0000000000000008:false:8...@1281058015446,000000000000000a:false:8...@1281058015447,000000000000000c:false:8...@1281058015447,000000000000000e:false:8...@1281058015447,0000000000000011:false:8...@1281058015447,0000000000000014:false:8...@1281058015447,0000000000000018:false:8...@1281058015447,000000000000001d:false:8...@1281058015447,0000000000000023:false:8...@1281058015447,000000000000002a:false:8...@1281058015447,0000000000000032:false:8...@1281058015447,000000000000003c:false:8...@1281058015447,0000000000000048:false:8...@1281058015447,0000000000000056:false:8...@1281058015447,0000000000000067:false:8...@1281058015447,000000000000007c:false:8...@1281058015447,0000000000000095:false:8...@1281058015447,00000000000000b3:false:8...@1281058015447,00000000000000d7:false:8...@1281058015447,0000000000000102:false:8...@1281058015447,0000000000000136:false:8...@1281058015447,0000000000000174:false:8...@1281058015447,00000000000001be:false:8...@1281058015447,0000000000000217:false:8...@1281058015447,0000000000000282:false:8...@1281058015447,0000000000000302:false:8...@1281058015447,000000000000039c:false:8...@1281058015447,0000000000000455:false:8...@1281058015447,0000000000000533:false:8...@1281058015447,000000000000063d:false:8...@1281058015447,000000000000077c:false:8...@1281058015447,00000000000008fb:false:8...@1281058015447,0000000000000ac7:false:8...@1281058015447,0000000000000cef:false:8...@1281058015447,0000000000000f85:false:8...@1281058015447,00000000000012a0:false:8...@1281058015447,000000000000165a:false:8...@1281058015447,0000000000001ad2:false:8...@1281058015447,000000000000202f:false:8...@1281058015447,000000000000269f:false:8...@1281058015447,0000000000002e58:false:8...@1281058015447,000000000000379d:false:8...@1281058015447,00000000000042bc:false:8...@1281058015447,0000000000005015:false:8...@1281058015447,0000000000006019:false:8...@1281058015447,0000000000007351:false:8...@1281058015447,0000000000008a61:false:8...@1281058015447,000000000000a60e:false:8...@1281058015447,000000000000c744:false:8...@1281058015447,000000000000ef1e:false:8...@1281058015447,0000000000011ef1:false:8...@1281058015447,0000000000015854:false:8...@1281058015447,0000000000019d32:false:8...@1281058015447,000000000001efd6:false:8...@1281058015447,0000000000025301:false:8...@1281058015447,000000000002ca01:false:8...@1281058015447,00000000000358ce:false:8...@1281058015447,000000000004042a:false:8...@1281058015447,000000000004d1cc:false:8...@1281058015447,000000000005c88e:false:8...@1281058015447,000000000006f0aa:false:8...@1281058015447,00000000000853ff:false:8...@1281058015447,000000000009fe65:false:8...@1281058015447,00000000000bfe13:false:8...@1281058015447,00000000000e6417:false:8...@1281058015447,00000000001144e8:false:8...@1281058015447,000000000014b916:false:8...@1281058015447,000000000018de1a:false:8...@1281058015447,00000000001dd752:false:8...@1281058015447,000000000023cf2f:false:8...@1281058015447,00000000002af89f:false:8...@1281058015447,00000000003390bf:false:8...@1281058015447,00000000003de0e5:false:8...@1281058015447,00000000004a4113:false:8...@1281058015447,0000000000591ae4:false:8...@1281058015447,00000000006aed12:false:8...@1281058015447,0000000000804faf:false:8...@1281058015447,000000000099f938:false:8...@1281058015447,0000000000b8c4aa:false:8...@1281058015448,0000000000ddb8cc:false:8...@1281058015448,00000000010a10f5:false:8...@1281058015448,00000000013f478c:false:8...@1281058015448,00000000017f22a8:false:8...@1281058015448,0000000001cbc330:false:8...@1281058015448,000000000227b706:false:8...@1281058015448,0000000002960ed4:false:8...@1281058015448,00000000031a7832:false:8...@1281058015448,0000000003b95d09:false:8...@1281058015448,000000000478093e:false:8...@1281058015448,00000000055cd7e4:false:8...@1281058015448,00000000066f6978:false:8...@1281058015448,0000000007b8e4f6:false:8...@1281058015448,00000000094445f4:false:8...@1281058015448,000000000b1eba58:false:8...@1281058015448,000000000d5812d0:false:8...@1281058015448,00000000100349c6:false:8...@1281058015448,0000000013372554:false:8...@1281058015448,00000000170ef998:false:8...@1281058015448,000000001bab91ea:false:8...@1281058015448,00000000213448b2:false:8...@1281058015448,0000000027d8573c:false:8...@1281058015448,000000002fd068ae:false:8...@1281058015448,0000000039607d9e:false:8...@1281058015448,0000000044da3057:false:8...@1281058015448,00000000529f6d35:false:8...@1281058015448,]),SuperColumn(RowSize
>
> [0000000000000001:false:8...@1281058015445,0000000000000002:false:8...@1281058015445,0000000000000003:false:8...@1281058015445,0000000000000004:false:8...@1281058015445,0000000000000005:false:8...@1281058015445,0000000000000006:false:8...@1281058015445,0000000000000007:false:8...@1281058015445,0000000000000008:false:8...@1281058015445,000000000000000a:false:8...@1281058015445,000000000000000c:false:8...@1281058015445,000000000000000e:false:8...@1281058015445,0000000000000011:false:8...@1281058015445,0000000000000014:false:8...@1281058015445,0000000000000018:false:8...@1281058015445,000000000000001d:false:8...@1281058015445,0000000000000023:false:8...@1281058015445,000000000000002a:false:8...@1281058015445,0000000000000032:false:8...@1281058015445,000000000000003c:false:8...@1281058015445,0000000000000048:false:8...@1281058015445,0000000000000056:false:8...@1281058015445,0000000000000067:false:8...@1281058015445,000000000000007c:false:8...@1281058015445,0000000000000095:false:8...@1281058015445,00000000000000b3:false:8...@1281058015445,00000000000000d7:false:8...@1281058015445,0000000000000102:false:8...@1281058015445,0000000000000136:false:8...@1281058015445,0000000000000174:false:8...@1281058015445,00000000000001be:false:8...@1281058015445,0000000000000217:false:8...@1281058015445,0000000000000282:false:8...@1281058015445,0000000000000302:false:8...@1281058015445,000000000000039c:false:8...@1281058015445,0000000000000455:false:8...@1281058015445,0000000000000533:false:8...@1281058015445,000000000000063d:false:8...@1281058015445,000000000000077c:false:8...@1281058015445,00000000000008fb:false:8...@1281058015445,0000000000000ac7:false:8...@1281058015445,0000000000000cef:false:8...@1281058015445,0000000000000f85:false:8...@1281058015445,00000000000012a0:false:8...@1281058015446,000000000000165a:false:8...@1281058015446,0000000000001ad2:false:8...@1281058015446,000000000000202f:false:8...@1281058015446,000000000000269f:false:8...@1281058015446,0000000000002e58:false:8...@1281058015446,000000000000379d:false:8...@1281058015446,00000000000042bc:false:8...@1281058015446,0000000000005015:false:8...@1281058015446,0000000000006019:false:8...@1281058015446,0000000000007351:false:8...@1281058015446,0000000000008a61:false:8...@1281058015446,000000000000a60e:false:8...@1281058015446,000000000000c744:false:8...@1281058015446,000000000000ef1e:false:8...@1281058015446,0000000000011ef1:false:8...@1281058015446,0000000000015854:false:8...@1281058015446,0000000000019d32:false:8...@1281058015446,000000000001efd6:false:8...@1281058015446,0000000000025301:false:8...@1281058015446,000000000002ca01:false:8...@1281058015446,00000000000358ce:false:8...@1281058015446,000000000004042a:false:8...@1281058015446,000000000004d1cc:false:8...@1281058015446,000000000005c88e:false:8...@1281058015446,000000000006f0aa:false:8...@1281058015446,00000000000853ff:false:8...@1281058015446,000000000009fe65:false:8...@1281058015446,00000000000bfe13:false:8...@1281058015446,00000000000e6417:false:8...@1281058015446,00000000001144e8:false:8...@1281058015446,000000000014b916:false:8...@1281058015446,000000000018de1a:false:8...@1281058015446,00000000001dd752:false:8...@1281058015446,000000000023cf2f:false:8...@1281058015446,00000000002af89f:false:8...@1281058015446,00000000003390bf:false:8...@1281058015446,00000000003de0e5:false:8...@1281058015446,00000000004a4113:false:8...@1281058015446,0000000000591ae4:false:8...@1281058015446,00000000006aed12:false:8...@1281058015446,0000000000804faf:false:8...@1281058015446,000000000099f938:false:8...@1281058015446,0000000000b8c4aa:false:8...@1281058015446,0000000000ddb8cc:false:8...@1281058015446,00000000010a10f5:false:8...@1281058015446,00000000013f478c:false:8...@1281058015446,00000000017f22a8:false:8...@1281058015446,0000000001cbc330:false:8...@1281058015446,000000000227b706:false:8...@1281058015446,0000000002960ed4:false:8...@1281058015446,00000000031a7832:false:8...@1281058015446,0000000003b95d09:false:8...@1281058015446,000000000478093e:false:8...@1281058015446,00000000055cd7e4:false:8...@1281058015446,00000000066f6978:false:8...@1281058015446,0000000007b8e4f6:false:8...@1281058015446,00000000094445f4:false:8...@1281058015446,000000000b1eba58:false:8...@1281058015446,000000000d5812d0:false:8...@1281058015446,00000000100349c6:false:8...@1281058015446,0000000013372554:false:8...@1281058015446,00000000170ef998:false:8...@1281058015446,000000001bab91ea:false:8...@1281058015446,00000000213448b2:false:8...@1281058015446,0000000027d8573c:false:8...@1281058015446,000000002fd068ae:false:8...@1281058015446,0000000039607d9e:false:8...@1281058015446,0000000044da3057:false:8...@1281058015446,00000000529f6d35:false:8...@1281058015446,000000006325b640:false:8...@1281058015446,0000000076fa0de6:false:8...@1281058015446,000000008ec5aa47:false:8...@1281058015446,00000000ab539922:false:8...@1281058015446,00000000cd97848f:false:8...@1281058015446,00000000f6b5d245:false:8...@1281058015446,00000001280d62b9:false:8...@1281058015446,0000000163434344:false:8...@1281058015446,00000001aa50b71e:false:8...@1281058015446,00000001ff940ef1:false:8...@1281058015446,0000000265e4debb:false:8...@1281058015446,00000002e0ac3e7a:false:8...@1281058015446,000000037401e492:false:8...@1281058015446,0000000424cf1249:false:8...@1281058015446,00000004f8f87c58:false:8...@1281058015446,00000005f7909536:false:8...@1281058015446,000000072913e641:false:8...@1281058015446,0000000897b17ab4:false:8...@1281058015446,]),])}
> INFO 18:32:08,817 Finished reading
> /var/lib/cassandra/commitlog/CommitLog-1281057918124.log
> DEBUG 18:32:08,818 Finished waiting on mutations from recovery
> INFO 18:32:08,819 Creating new commitlog segment
> /var/lib/cassandra/commitlog/CommitLog-1281058328819.log
> INFO 18:32:08,847 switching in a fresh Memtable for Statistics at
> CommitLogContext(file='/var/lib/cassandra/commitlog/CommitLog-1281058328819.log',
> position=0)
> INFO 18:32:08,856 Enqueuing flush of memtable-statist...@713462035(7502
> bytes, 2 operations)
> INFO 18:32:08,857 Writing memtable-statist...@713462035(7502 bytes, 2
> operations)
> INFO 18:32:08,961 Completed flushing
> /var/lib/cassandra/data/system/Statistics-e-2-Data.db
> DEBUG 18:32:08,962 key cache capacity for Statistics is 2
> DEBUG 18:32:08,966 Checking to see if compaction of Statistics would be useful
> DEBUG 18:32:08,966 Discarding 4
> DEBUG 18:32:08,968 discard completed log segments for
> CommitLogContext(file='/var/lib/cassandra/commitlog/CommitLog-1281058328819.log',
> position=0), column family 4.
> DEBUG 18:32:08,968 Marking replay position 0 on commit log
> CommitLogSegment(/var/lib/cassandra/commitlog/CommitLog-1281058328819.log)
> INFO 18:32:08,969 Recovery complete
> DEBUG 18:32:08,970 Deleting CommitLog-1281057874925.log
> DEBUG 18:32:08,971 Deleting CommitLog-1281057918124.log
> INFO 18:32:08,974 Log replay complete
> DEBUG 18:32:08,980 Estimating compactions for Super1
> DEBUG 18:32:08,980 Estimating compactions for Standard1
> DEBUG 18:32:08,981 Estimating compactions for LocationInfo
> DEBUG 18:32:08,982 Estimating compactions for HintsColumnFamily
> DEBUG 18:32:08,982 Estimating compactions for Migrations
> DEBUG 18:32:08,982 Estimating compactions for Schema
> DEBUG 18:32:08,982 Estimating compactions for Statistics
> INFO 18:32:08,983 Cassandra version: 0.7.0-SNAPSHOT
> INFO 18:32:08,983 Thrift API version: 10.0.0
> DEBUG 18:32:08,983 Checking to see if compaction of Super1 would be useful
> DEBUG 18:32:08,983 Checking to see if compaction of Standard1 would be useful
> INFO 18:32:08,984 Compacting
> [org.apache.cassandra.io.sstable.SSTableReader(path='/var/lib/cassandra/data/Keyspace1/Standard1-e-32-Data.db'),org.apache.cassandra.io.sstable.SSTableReader(path='/var/lib/cassandra/data/Keyspace1/Standard1-e-33-Data.db'),org.apache.cassandra.io.sstable.SSTableReader(path='/var/lib/cassandra/data/Keyspace1/Standard1-e-34-Data.db'),org.apache.cassandra.io.sstable.SSTableReader(path='/var/lib/cassandra/data/Keyspace1/Standard1-e-35-Data.db'),org.apache.cassandra.io.sstable.SSTableReader(path='/var/lib/cassandra/data/Keyspace1/Standard1-e-36-Data.db')]
> INFO 18:32:08,984 Saved Token found: 43976851516388127359611406752167831045
> DEBUG 18:32:08,984 index size for bloom filter calc for file :
> /var/lib/cassandra/data/Keyspace1/Standard1-e-32-Data.db : 10624
> INFO 18:32:08,985 Saved ClusterName found: Test Cluster
> DEBUG 18:32:08,985 index size for bloom filter calc for file :
> /var/lib/cassandra/data/Keyspace1/Standard1-e-33-Data.db : 11264
> DEBUG 18:32:08,985 index size for bloom filter calc for file :
> /var/lib/cassandra/data/Keyspace1/Standard1-e-34-Data.db : 27136
> DEBUG 18:32:08,985 index size for bloom filter calc for file :
> /var/lib/cassandra/data/Keyspace1/Standard1-e-35-Data.db : 43008
> DEBUG 18:32:08,985 index size for bloom filter calc for file :
> /var/lib/cassandra/data/Keyspace1/Standard1-e-36-Data.db : 57984
> DEBUG 18:32:08,985 Expected bloom filter size : 57984
> INFO 18:32:08,985 Saved partitioner not found. Using
> org.apache.cassandra.dht.RandomPartitioner
> INFO 18:32:08,988 switching in a fresh Memtable for LocationInfo at
> CommitLogContext(file='/var/lib/cassandra/commitlog/CommitLog-1281058328819.log',
> position=276)
> INFO 18:32:08,988 Enqueuing flush of memtable-locationi...@122395252(95
> bytes, 2 operations)
> INFO 18:32:08,989 Writing memtable-locationi...@122395252(95 bytes, 2
> operations)
> ERROR 18:32:09,041 Uncaught exception in thread
> Thread[CompactionExecutor:1,5,main]
> java.util.concurrent.ExecutionException: java.lang.NoSuchMethodError:
> org.apache.cassandra.config.DatabaseDescriptor.getInMemoryCompactionLimit()J
> at java.util.concurrent.FutureTask$Sync.innerGet(FutureTask.java:222)
> at java.util.concurrent.FutureTask.get(FutureTask.java:83)
> at
> org.apache.cassandra.concurrent.DebuggableThreadPoolExecutor.afterExecute(DebuggableThreadPoolExecutor.java:87)
> at
> org.apache.cassandra.db.CompactionManager$CompactionExecutor.afterExecute(CompactionManager.java:650)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:888)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> at java.lang.Thread.run(Thread.java:637)
> Caused by: java.lang.NoSuchMethodError:
> org.apache.cassandra.config.DatabaseDescriptor.getInMemoryCompactionLimit()J
> at
> org.apache.cassandra.io.CompactionIterator.getCompactedRow(CompactionIterator.java:132)
> at
> org.apache.cassandra.io.CompactionIterator.getReduced(CompactionIterator.java:107)
> at
> org.apache.cassandra.io.CompactionIterator.getReduced(CompactionIterator.java:46)
> at
> org.apache.cassandra.utils.ReducingIterator.computeNext(ReducingIterator.java:73)
> at
> com.google.common.collect.AbstractIterator.tryToComputeNext(AbstractIterator.java:136)
> at
> com.google.common.collect.AbstractIterator.hasNext(AbstractIterator.java:131)
> at
> org.apache.commons.collections.iterators.FilterIterator.setNextObject(FilterIterator.java:183)
> at
> org.apache.commons.collections.iterators.FilterIterator.hasNext(FilterIterator.java:94)
> at
> org.apache.cassandra.db.CompactionManager.doCompaction(CompactionManager.java:333)
> at
> org.apache.cassandra.db.CompactionManager$2.call(CompactionManager.java:151)
> at
> org.apache.cassandra.db.CompactionManager$2.call(CompactionManager.java:132)
> at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
> at java.util.concurrent.FutureTask.run(FutureTask.java:138)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> ... 2 more
> DEBUG 18:32:09,044 Checking to see if compaction of LocationInfo would be
> useful
> INFO 18:32:09,044 Compacting
> [org.apache.cassandra.io.sstable.SSTableReader(path='/var/lib/cassandra/data/system/LocationInfo-e-1-Data.db'),org.apache.cassandra.io.sstable.SSTableReader(path='/var/lib/cassandra/data/system/LocationInfo-e-2-Data.db'),org.apache.cassandra.io.sstable.SSTableReader(path='/var/lib/cassandra/data/system/LocationInfo-e-3-Data.db'),org.apache.cassandra.io.sstable.SSTableReader(path='/var/lib/cassandra/data/system/LocationInfo-e-4-Data.db')]
> DEBUG 18:32:09,044 index size for bloom filter calc for file :
> /var/lib/cassandra/data/system/LocationInfo-e-1-Data.db : 256
> DEBUG 18:32:09,044 index size for bloom filter calc for file :
> /var/lib/cassandra/data/system/LocationInfo-e-2-Data.db : 512
> DEBUG 18:32:09,045 index size for bloom filter calc for file :
> /var/lib/cassandra/data/system/LocationInfo-e-3-Data.db : 768
> DEBUG 18:32:09,045 index size for bloom filter calc for file :
> /var/lib/cassandra/data/system/LocationInfo-e-4-Data.db : 1024
> DEBUG 18:32:09,045 Expected bloom filter size : 1024
> ERROR 18:32:09,048 Uncaught exception in thread
> Thread[CompactionExecutor:1,5,main]
> java.util.concurrent.ExecutionException: java.lang.NoSuchMethodError:
> org.apache.cassandra.config.DatabaseDescriptor.getInMemoryCompactionLimit()J
> at java.util.concurrent.FutureTask$Sync.innerGet(FutureTask.java:222)
> at java.util.concurrent.FutureTask.get(FutureTask.java:83)
> at
> org.apache.cassandra.concurrent.DebuggableThreadPoolExecutor.afterExecute(DebuggableThreadPoolExecutor.java:87)
> at
> org.apache.cassandra.db.CompactionManager$CompactionExecutor.afterExecute(CompactionManager.java:650)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:888)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> at java.lang.Thread.run(Thread.java:637)
> Caused by: java.lang.NoSuchMethodError:
> org.apache.cassandra.config.DatabaseDescriptor.getInMemoryCompactionLimit()J
> at
> org.apache.cassandra.io.CompactionIterator.getCompactedRow(CompactionIterator.java:132)
> at
> org.apache.cassandra.io.CompactionIterator.getReduced(CompactionIterator.java:107)
> at
> org.apache.cassandra.io.CompactionIterator.getReduced(CompactionIterator.java:46)
> at
> org.apache.cassandra.utils.ReducingIterator.computeNext(ReducingIterator.java:73)
> at
> com.google.common.collect.AbstractIterator.tryToComputeNext(AbstractIterator.java:136)
> at
> com.google.common.collect.AbstractIterator.hasNext(AbstractIterator.java:131)
> at
> org.apache.commons.collections.iterators.FilterIterator.setNextObject(FilterIterator.java:183)
> at
> org.apache.commons.collections.iterators.FilterIterator.hasNext(FilterIterator.java:94)
> at
> org.apache.cassandra.db.CompactionManager.doCompaction(CompactionManager.java:333)
> at
> org.apache.cassandra.db.CompactionManager$2.call(CompactionManager.java:151)
> at
> org.apache.cassandra.db.CompactionManager$2.call(CompactionManager.java:132)
> at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
> at java.util.concurrent.FutureTask.run(FutureTask.java:138)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> ... 2 more
> DEBUG 18:32:09,051 Checking to see if compaction of HintsColumnFamily would
> be useful
> DEBUG 18:32:09,052 Checking to see if compaction of Migrations would be useful
> DEBUG 18:32:09,052 Checking to see if compaction of Schema would be useful
> DEBUG 18:32:09,052 Checking to see if compaction of Statistics would be useful
> INFO 18:32:09,108 Completed flushing
> /var/lib/cassandra/data/system/LocationInfo-e-5-Data.db
> DEBUG 18:32:09,109 key cache capacity for LocationInfo is 6
> DEBUG 18:32:09,110 Checking to see if compaction of LocationInfo would be
> useful
> DEBUG 18:32:09,110 Discarding 0
> INFO 18:32:09,111 Compacting
> [org.apache.cassandra.io.sstable.SSTableReader(path='/var/lib/cassandra/data/system/LocationInfo-e-1-Data.db'),org.apache.cassandra.io.sstable.SSTableReader(path='/var/lib/cassandra/data/system/LocationInfo-e-2-Data.db'),org.apache.cassandra.io.sstable.SSTableReader(path='/var/lib/cassandra/data/system/LocationInfo-e-3-Data.db'),org.apache.cassandra.io.sstable.SSTableReader(path='/var/lib/cassandra/data/system/LocationInfo-e-4-Data.db'),org.apache.cassandra.io.sstable.SSTableReader(path='/var/lib/cassandra/data/system/LocationInfo-e-5-Data.db')]
> DEBUG 18:32:09,111 discard completed log segments for
> CommitLogContext(file='/var/lib/cassandra/commitlog/CommitLog-1281058328819.log',
> position=276), column family 0.
> DEBUG 18:32:09,111 Marking replay position 276 on commit log
> CommitLogSegment(/var/lib/cassandra/commitlog/CommitLog-1281058328819.log)
> DEBUG 18:32:09,111 index size for bloom filter calc for file :
> /var/lib/cassandra/data/system/LocationInfo-e-1-Data.db : 256
> DEBUG 18:32:09,112 index size for bloom filter calc for file :
> /var/lib/cassandra/data/system/LocationInfo-e-2-Data.db : 512
> DEBUG 18:32:09,112 index size for bloom filter calc for file :
> /var/lib/cassandra/data/system/LocationInfo-e-3-Data.db : 768
> DEBUG 18:32:09,112 index size for bloom filter calc for file :
> /var/lib/cassandra/data/system/LocationInfo-e-4-Data.db : 1024
> DEBUG 18:32:09,112 index size for bloom filter calc for file :
> /var/lib/cassandra/data/system/LocationInfo-e-5-Data.db : 1280
> DEBUG 18:32:09,112 Expected bloom filter size : 1280
> ERROR 18:32:09,123 Uncaught exception in thread
> Thread[CompactionExecutor:1,5,main]
> java.util.concurrent.ExecutionException: java.lang.NoSuchMethodError:
> org.apache.cassandra.config.DatabaseDescriptor.getInMemoryCompactionLimit()J
> at java.util.concurrent.FutureTask$Sync.innerGet(FutureTask.java:222)
> at java.util.concurrent.FutureTask.get(FutureTask.java:83)
> at
> org.apache.cassandra.concurrent.DebuggableThreadPoolExecutor.afterExecute(DebuggableThreadPoolExecutor.java:87)
> at
> org.apache.cassandra.db.CompactionManager$CompactionExecutor.afterExecute(CompactionManager.java:650)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:888)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> at java.lang.Thread.run(Thread.java:637)
> Caused by: java.lang.NoSuchMethodError:
> org.apache.cassandra.config.DatabaseDescriptor.getInMemoryCompactionLimit()J
> at
> org.apache.cassandra.io.CompactionIterator.getCompactedRow(CompactionIterator.java:132)
> at
> org.apache.cassandra.io.CompactionIterator.getReduced(CompactionIterator.java:107)
> at
> org.apache.cassandra.io.CompactionIterator.getReduced(CompactionIterator.java:46)
> at
> org.apache.cassandra.utils.ReducingIterator.computeNext(ReducingIterator.java:73)
> at
> com.google.common.collect.AbstractIterator.tryToComputeNext(AbstractIterator.java:136)
> at
> com.google.common.collect.AbstractIterator.hasNext(AbstractIterator.java:131)
> at
> org.apache.commons.collections.iterators.FilterIterator.setNextObject(FilterIterator.java:183)
> at
> org.apache.commons.collections.iterators.FilterIterator.hasNext(FilterIterator.java:94)
> at
> org.apache.cassandra.db.CompactionManager.doCompaction(CompactionManager.java:333)
> at
> org.apache.cassandra.db.CompactionManager$2.call(CompactionManager.java:151)
> at
> org.apache.cassandra.db.CompactionManager$2.call(CompactionManager.java:132)
> at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
> at java.util.concurrent.FutureTask.run(FutureTask.java:138)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> ... 2 more
> INFO 18:32:09,167 Starting up server gossip
> DEBUG 18:32:09,248 attempting to connect to /127.0.0.1
> DEBUG 18:32:09,252 clearing cached endpoints
> DEBUG 18:32:09,252 clearing cached endpoints
> DEBUG 18:32:09,252 attempting to connect to /192.168.101.51
> INFO 18:32:09,276 Binding thrift service to /192.168.101.168:9160
> INFO 18:32:09,278 Using TFramedTransport with a max frame size of 15728640
> bytes.
> INFO 18:32:09,280 Listening for thrift clients...
> DEBUG 18:32:09,280 Will try to load mx4j now, if it's in the classpath
> DEBUG 18:32:09,322 attempting to connect to /192.168.101.51
> INFO 18:32:09,375 Node /192.168.101.51 is now part of the cluster
> DEBUG 18:32:09,375 Resetting pool for /192.168.101.51
> DEBUG 18:32:09,377 Node /192.168.101.51 state normal, token
> 135629853287756932550538670294184981659
> DEBUG 18:32:09,377 clearing cached endpoints
> DEBUG 18:32:09,378 clearing cached endpoints
> DEBUG 18:32:09,378 No bootstrapping or leaving nodes -> empty pending ranges
> for Keyspace1
> INFO 18:32:09,381 mx4j successfuly loaded
> HttpAdaptor version 3.0.2 started on port 8081
> DEBUG 18:32:09,803 attempting to connect to /192.168.101.51
> INFO 18:32:09,810 Started hinted handoff for endpoint /192.168.101.51
> INFO 18:32:09,810 InetAddress /192.168.101.51 is now UP
> INFO 18:32:09,811 Finished hinted handoff of 0 rows to endpoint
> /192.168.101.51
> DEBUG 18:32:10,175 GC for ParNew: 5 ms, 16210960 reclaimed leaving 193987168
> used; max is 1207828480
> DEBUG 18:32:11,229 attempting to connect to /127.0.0.1
> DEBUG 18:32:11,244 Disseminating load info ...
> {noformat}
> 5) After some restarts the error changes into:
> {noformat}
> INFO 18:32:40,951 Replaying
> /var/lib/cassandra/commitlog/CommitLog-1281058340642.log
> DEBUG 18:32:40,953 Replaying
> /var/lib/cassandra/commitlog/CommitLog-1281058340642.log starting at 276
> DEBUG 18:32:40,953 Reading mutation at 276
> DEBUG 18:32:40,956 replaying mutation for system...@77fe4169:
> {ColumnFamily(LocationInfo [B:false:1...@1281058340821,])}
> DEBUG 18:32:40,965 Reading mutation at 424
> INFO 18:32:40,966 Finished reading
> /var/lib/cassandra/commitlog/CommitLog-1281058340642.log
> ERROR 18:32:40,967 Exception encountered during startup.
> org.apache.cassandra.db.marshal.MarshalException: invalid UTF8 bytes [-64,
> -88, 101, 51]
> at org.apache.cassandra.db.marshal.UTF8Type.getString(UTF8Type.java:43)
> at org.apache.cassandra.db.Column.getString(Column.java:247)
> at
> org.apache.cassandra.db.marshal.AbstractType.getColumnsString(AbstractType.java:85)
> at org.apache.cassandra.db.ColumnFamily.toString(ColumnFamily.java:379)
> at org.apache.commons.lang.ObjectUtils.toString(ObjectUtils.java:241)
> at org.apache.commons.lang.StringUtils.join(StringUtils.java:3073)
> at org.apache.commons.lang.StringUtils.join(StringUtils.java:3133)
> at
> org.apache.cassandra.db.commitlog.CommitLog.recover(CommitLog.java:279)
> at
> org.apache.cassandra.db.commitlog.CommitLog.recover(CommitLog.java:174)
> at
> org.apache.cassandra.thrift.CassandraDaemon.setup(CassandraDaemon.java:120)
> at
> org.apache.cassandra.service.AbstractCassandraDaemon.activate(AbstractCassandraDaemon.java:90)
> at
> org.apache.cassandra.thrift.CassandraDaemon.main(CassandraDaemon.java:224)
> Exception encountered during startup.
> org.apache.cassandra.db.marshal.MarshalException: invalid UTF8 bytes [-64,
> -88, 101, 51]
> at org.apache.cassandra.db.marshal.UTF8Type.getString(UTF8Type.java:43)
> at org.apache.cassandra.db.Column.getString(Column.java:247)
> at
> org.apache.cassandra.db.marshal.AbstractType.getColumnsString(AbstractType.java:85)
> at org.apache.cassandra.db.ColumnFamily.toString(ColumnFamily.java:379)
> at org.apache.commons.lang.ObjectUtils.toString(ObjectUtils.java:241)
> at org.apache.commons.lang.StringUtils.join(StringUtils.java:3073)
> at org.apache.commons.lang.StringUtils.join(StringUtils.java:3133)
> at
> org.apache.cassandra.db.commitlog.CommitLog.recover(CommitLog.java:279)
> at
> org.apache.cassandra.db.commitlog.CommitLog.recover(CommitLog.java:174)
> at
> org.apache.cassandra.thrift.CassandraDaemon.setup(CassandraDaemon.java:120)
> at
> org.apache.cassandra.service.AbstractCassandraDaemon.activate(AbstractCassandraDaemon.java:90)
> at
> org.apache.cassandra.thrift.CassandraDaemon.main(CassandraDaemon.java:224)
> {noformat}
> 6) Then bring down cassandra on the side that is raising errors. Cleanup and
> keep autobootstrap=false. Put datafiles of the snapshot in
> /var/lib/cassandra/data/Keyspace1/, start cassandra, it gossips the schema
> and picks up some of the files but doesn't replay the log apparently so no
> errors are displayed. Then stop and restart and while replaying the log I see
> those errors again.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
