You may have a corrupt metadata/statistics sstable component. You can try deleting those and restarting. Cassandra can rebuild that component if it is missing.
On Fri, Jul 6, 2012 at 6:00 PM, Jason Hill <jasonhill...@gmail.com> wrote: > Hello friends, > > I'm getting a: > > ERROR 22:50:29,695 Fatal exception in thread Thread[SSTableBatchOpen:2,5,main] > java.lang.OutOfMemoryError: Java heap space > > error when I start Cassandra. This node was running fine and after > some server work/upgrades it started throwing this error when I start > the Cassandra service. I was on 0.8.? and have upgraded to 1.0.10 to > see if it would help, but I get the same error. I've removed some of > the column families from my keyspace directory to see if I can get it > to start without the heap space error and with some combinations it > will run. However, I'd like to get it running with all my colFams and > wonder if someone could give me some advice on what might be causing > my error. It doesn't seem to be related to compaction, if I am reading > the log correctly, and most of the help I've found on this topic deals > with compaction. I'm thinking that my 2 column families should not be > enough to fill my heap, but I am at a loss as to what I should try > next? > > Thanks for your consideration. > > output.log: > > INFO 22:50:26,319 JVM vendor/version: Java HotSpot(TM) 64-Bit Server > VM/1.6.0_26 > INFO 22:50:26,322 Heap size: 5905580032/5905580032 > INFO 22:50:26,322 Classpath: > /usr/share/cassandra/lib/antlr-3.2.jar:/usr/share/cassandra/lib/avro-1.4.0-fixes.jar:/usr/share/cassandra/lib/avro-1.4.0-sources-fixes.jar:/usr/share/cassandra/lib/commons-cli-1.1.jar:/usr/share/cassandra/lib/commons-codec-1.2.jar:/usr/share/cassandra/lib/commons-lang-2.4.jar:/usr/share/cassandra/lib/compress-lzf-0.8.4.jar:/usr/share/cassandra/lib/concurrentlinkedhashmap-lru-1.2.jar:/usr/share/cassandra/lib/guava-r08.jar:/usr/share/cassandra/lib/high-scale-lib-1.1.2.jar:/usr/share/cassandra/lib/jackson-core-asl-1.4.0.jar:/usr/share/cassandra/lib/jackson-mapper-asl-1.4.0.jar:/usr/share/cassandra/lib/jamm-0.2.5.jar:/usr/share/cassandra/lib/jline-0.9.94.jar:/usr/share/cassandra/lib/json-simple-1.1.jar:/usr/share/cassandra/lib/libthrift-0.6.jar:/usr/share/cassandra/lib/log4j-1.2.16.jar:/usr/share/cassandra/lib/servlet-api-2.5-20081211.jar:/usr/share/cassandra/lib/slf4j-api-1.6.1.jar:/usr/share/cassandra/lib/slf4j-log4j12-1.6.1.jar:/usr/share/cassandra/lib/snakeyaml-1.6.jar:/usr/share/cassandra/lib/snappy-java-1.0.4.1.jar:/usr/share/cassandra/apache-cassandra-1.0.10.jar:/usr/share/cassandra/apache-cassandra-thrift-1.0.10.jar:/usr/share/cassandra/apache-cassandra.jar:/usr/share/java/jna.jar:/etc/cassandra:/usr/share/java/commons-daemon.jar:/usr/share/cassandra/lib/jamm-0.2.5.jar > INFO 22:50:28,586 JNA mlockall successful > INFO 22:50:28,593 Loading settings from file:/etc/cassandra/cassandra.yaml > DEBUG 22:50:28,677 Syncing log with a period of 10000 > INFO 22:50:28,677 DiskAccessMode 'auto' determined to be mmap, > indexAccessMode is mmap > INFO 22:50:28,686 Global memtable threshold is enabled at 1877MB > DEBUG 22:50:28,761 setting auto_bootstrap to true > <snip> > DEBUG 22:50:28,797 Checking directory /var/lib/cassandra/data > DEBUG 22:50:28,798 Checking directory /var/lib/cassandra/commitlog > DEBUG 22:50:28,798 Checking directory /var/lib/cassandra/saved_caches > DEBUG 22:50:28,806 Removing compacted SSTable files from NodeIdInfo > (see http://wiki.apache.org/cassandra/MemtableSSTable) > DEBUG 22:50:28,808 Removing compacted SSTable files from Versions (see > http://wiki.apache.org/cassandra/MemtableSSTable) > DEBUG 22:50:28,818 Removing compacted SSTable files from > Versions.76657273696f6e (see > http://wiki.apache.org/cassandra/MemtableSSTable) > DEBUG 22:50:28,819 Removing compacted SSTable files from IndexInfo > (see http://wiki.apache.org/cassandra/MemtableSSTable) > DEBUG 22:50:28,821 Removing compacted SSTable files from Schema (see > http://wiki.apache.org/cassandra/MemtableSSTable) > DEBUG 22:50:28,823 Removing compacted SSTable files from Migrations > (see http://wiki.apache.org/cassandra/MemtableSSTable) > DEBUG 22:50:28,825 Removing compacted SSTable files from LocationInfo > (see http://wiki.apache.org/cassandra/MemtableSSTable) > DEBUG 22:50:28,827 Removing compacted SSTable files from > HintsColumnFamily (see > http://wiki.apache.org/cassandra/MemtableSSTable) > DEBUG 22:50:28,833 Initializing system.NodeIdInfo > DEBUG 22:50:28,839 Starting CFS NodeIdInfo > DEBUG 22:50:28,868 Creating IntervalNode from [] > DEBUG 22:50:28,869 KeyCache capacity for NodeIdInfo is 1 > DEBUG 22:50:28,871 Initializing system.Versions > DEBUG 22:50:28,873 Starting CFS Versions > INFO 22:50:28,877 Opening > /var/lib/cassandra/data/system/Versions-hd-5 (248 bytes) > DEBUG 22:50:28,879 Load metadata for > /var/lib/cassandra/data/system/Versions-hd-5 > INFO 22:50:28,880 Opening > /var/lib/cassandra/data/system/Versions-hd-6 (248 bytes) > DEBUG 22:50:28,880 Load metadata for > /var/lib/cassandra/data/system/Versions-hd-6 > DEBUG 22:50:28,897 INDEX LOAD TIME for > /var/lib/cassandra/data/system/Versions-hd-5: 20 ms. > DEBUG 22:50:28,898 INDEX LOAD TIME for > /var/lib/cassandra/data/system/Versions-hd-6: 18 ms. > DEBUG 22:50:28,899 key cache contains 0/0 keys > DEBUG 22:50:28,899 key cache contains 0/0 keys > INFO 22:50:28,899 Opening > /var/lib/cassandra/data/system/Versions-hd-7 (248 bytes) > DEBUG 22:50:28,900 Load metadata for > /var/lib/cassandra/data/system/Versions-hd-7 > DEBUG 22:50:28,901 INDEX LOAD TIME for > /var/lib/cassandra/data/system/Versions-hd-7: 2 ms. > DEBUG 22:50:28,901 key cache contains 0/0 keys > DEBUG 22:50:28,903 Creating IntervalNode from > [Interval(DecoratedKey(93220794208128599841715671226150005828, > 746872696674), DecoratedKey(135285944860343992175601105924967452217, > 63716c)), Interval(DecoratedKey(93220794208128599841715671226150005828, > 746872696674), DecoratedKey(135285944860343992175601105924967452217, > 63716c)), Interval(DecoratedKey(93220794208128599841715671226150005828, > 746872696674), DecoratedKey(135285944860343992175601105924967452217, > 63716c))] > DEBUG 22:50:28,905 adding /var/lib/cassandra/data/system/Versions-hd-5 > to list of files tracked for system.Versions > DEBUG 22:50:28,905 adding /var/lib/cassandra/data/system/Versions-hd-6 > to list of files tracked for system.Versions > DEBUG 22:50:28,906 adding /var/lib/cassandra/data/system/Versions-hd-7 > to list of files tracked for system.Versions > DEBUG 22:50:28,906 KeyCache capacity for Versions is 3 > DEBUG 22:50:28,906 Initializing system.IndexInfo > DEBUG 22:50:28,908 Starting CFS IndexInfo > DEBUG 22:50:28,910 Creating IntervalNode from [] > DEBUG 22:50:28,910 KeyCache capacity for IndexInfo is 1 > DEBUG 22:50:28,910 Initializing system.Schema > DEBUG 22:50:28,912 Starting CFS Schema > INFO 22:50:28,915 Opening /var/lib/cassandra/data/system/Schema-hd-46 > (3831 bytes) > DEBUG 22:50:28,916 Load metadata for > /var/lib/cassandra/data/system/Schema-hd-46 > DEBUG 22:50:28,918 INDEX LOAD TIME for > /var/lib/cassandra/data/system/Schema-hd-46: 3 ms. > INFO 22:50:28,915 Opening /var/lib/cassandra/data/system/Schema-hd-45 > (187275 bytes) > DEBUG 22:50:28,918 Load metadata for > /var/lib/cassandra/data/system/Schema-hd-45 > DEBUG 22:50:28,918 key cache contains 0/0 keys > DEBUG 22:50:28,920 INDEX LOAD TIME for > /var/lib/cassandra/data/system/Schema-hd-45: 5 ms. > DEBUG 22:50:28,921 key cache contains 0/0 keys > DEBUG 22:50:28,922 Creating IntervalNode from > [Interval(DecoratedKey(124804735337540159479107746638263794797, > 4c617374204d6967726174696f6e), > DecoratedKey(168749730613100185197104044460479972233, > 62626139343461302d633734632d313165312d303030302d353662363634353961646666)), > Interval(DecoratedKey(2957058266746302108707016693211230853, > 64323438663637302d633438662d313165312d303030302d366637313036623962396666), > DecoratedKey(167756402517845603382936953195099512584, > 65306532376534302d633734622d313165312d303030302d353662363634353961646666))] > DEBUG 22:50:28,922 adding /var/lib/cassandra/data/system/Schema-hd-46 > to list of files tracked for system.Schema > DEBUG 22:50:28,923 adding /var/lib/cassandra/data/system/Schema-hd-45 > to list of files tracked for system.Schema > DEBUG 22:50:28,923 KeyCache capacity for Schema is 2 > DEBUG 22:50:28,924 Initializing system.Migrations > DEBUG 22:50:28,926 Starting CFS Migrations > INFO 22:50:28,929 Opening > /var/lib/cassandra/data/system/Migrations-hd-45 (323713 bytes) > DEBUG 22:50:28,930 Load metadata for > /var/lib/cassandra/data/system/Migrations-hd-45 > INFO 22:50:28,931 Opening > /var/lib/cassandra/data/system/Migrations-hd-46 (7945 bytes) > DEBUG 22:50:28,931 Load metadata for > /var/lib/cassandra/data/system/Migrations-hd-46 > DEBUG 22:50:28,932 INDEX LOAD TIME for > /var/lib/cassandra/data/system/Migrations-hd-45: 3 ms. > DEBUG 22:50:28,933 INDEX LOAD TIME for > /var/lib/cassandra/data/system/Migrations-hd-46: 2 ms. > DEBUG 22:50:28,933 key cache contains 0/0 keys > DEBUG 22:50:28,933 key cache contains 0/0 keys > DEBUG 22:50:28,935 Creating IntervalNode from > [Interval(DecoratedKey(159559397954378837828954138596956659794, > 4d6967726174696f6e73204b6579), > DecoratedKey(159559397954378837828954138596956659794, > 4d6967726174696f6e73204b6579)), > Interval(DecoratedKey(159559397954378837828954138596956659794, > 4d6967726174696f6e73204b6579), > DecoratedKey(159559397954378837828954138596956659794, > 4d6967726174696f6e73204b6579))] > DEBUG 22:50:28,936 adding > /var/lib/cassandra/data/system/Migrations-hd-45 to list of files > tracked for system.Migrations > DEBUG 22:50:28,937 adding > /var/lib/cassandra/data/system/Migrations-hd-46 to list of files > tracked for system.Migrations > DEBUG 22:50:28,937 KeyCache capacity for Migrations is 2 > DEBUG 22:50:28,938 Initializing system.LocationInfo > DEBUG 22:50:28,940 Starting CFS LocationInfo > INFO 22:50:28,943 Opening > /var/lib/cassandra/data/system/LocationInfo-hd-17 (357 bytes) > DEBUG 22:50:28,944 Load metadata for > /var/lib/cassandra/data/system/LocationInfo-hd-17 > DEBUG 22:50:28,947 INDEX LOAD TIME for > /var/lib/cassandra/data/system/LocationInfo-hd-17: 4 ms. > DEBUG 22:50:28,948 key cache contains 0/0 keys > DEBUG 22:50:28,948 Creating IntervalNode from > [Interval(DecoratedKey(61078635599166706937511052402724559481, 4c), > DecoratedKey(136380791207214638541884810421800860241, > 426f6f747374726170))] > DEBUG 22:50:28,949 adding > /var/lib/cassandra/data/system/LocationInfo-hd-17 to list of files > tracked for system.LocationInfo > DEBUG 22:50:28,949 KeyCache capacity for LocationInfo is 1 > DEBUG 22:50:28,949 Initializing system.HintsColumnFamily > DEBUG 22:50:28,951 Starting CFS HintsColumnFamily > DEBUG 22:50:28,953 Creating IntervalNode from [] > DEBUG 22:50:28,954 KeyCache capacity for HintsColumnFamily is 1 > DEBUG 22:50:28,959 collectTimeOrderedData > DEBUG 22:50:28,975 collectTimeOrderedData > INFO 22:50:28,977 Loading schema version bba944a0-c74c-11e1-0000-56b66459adff > DEBUG 22:50:28,978 collectAllData > DEBUG 22:50:28,983 collecting 0 of 2147483647: > Avro/Schema:false:2799@1341566817771 > DEBUG 22:50:28,983 collecting 1 of 2147483647: > Innovari:false:788@1341566817771 > DEBUG 22:50:29,143 Adding > org.apache.cassandra.config.CFMetaData@37e55794[cfId=1017,ksName=Innovari,cfName=Temps,cfType=Standard,comparator=org.apache.cassandra.db.marshal.LongType,subcolumncomparator=<null>,comment=,rowCacheSize=0.0,keyCacheSize=200000.0,readRepairChance=1.0,replicateOnWrite=true,gcGraceSeconds=864000,defaultValidator=org.apache.cassandra.db.marshal.BytesType,keyValidator=org.apache.cassandra.db.marshal.BytesType,minCompactionThreshold=4,maxCompactionThreshold=32,rowCacheSavePeriodInSeconds=0,keyCacheSavePeriodInSeconds=14400,rowCacheKeysToSave=2147483647,rowCacheProvider=org.apache.cassandra.cache.SerializingCacheProvider@31d8d237,mergeShardsChance=0.1,keyAlias=<null>,column_metadata={},compactionStrategyClass=class > org.apache.cassandra.db.compaction.SizeTieredCompactionStrategy,compactionStrategyOptions={},compressionOptions={},bloomFilterFpChance=<null>] > to cfIdMap > DEBUG 22:50:29,144 Adding > org.apache.cassandra.config.CFMetaData@3d46e381[cfId=1004,ksName=Innovari,cfName=TICStatusCurrent,cfType=Standard,comparator=org.apache.cassandra.db.marshal.AsciiType,subcolumncomparator=<null>,comment=,rowCacheSize=0.0,keyCacheSize=200000.0,readRepairChance=1.0,replicateOnWrite=true,gcGraceSeconds=864000,defaultValidator=org.apache.cassandra.db.marshal.BytesType,keyValidator=org.apache.cassandra.db.marshal.BytesType,minCompactionThreshold=4,maxCompactionThreshold=32,rowCacheSavePeriodInSeconds=0,keyCacheSavePeriodInSeconds=14400,rowCacheKeysToSave=2147483647,rowCacheProvider=org.apache.cassandra.cache.ConcurrentLinkedHashCacheProvider@79024994,mergeShardsChance=0.1,keyAlias=<null>,column_metadata={},compactionStrategyClass=class > org.apache.cassandra.db.compaction.SizeTieredCompactionStrategy,compactionStrategyOptions={},compressionOptions={},bloomFilterFpChance=<null>] > to cfIdMap > DEBUG 22:50:29,145 Removing compacted SSTable files from NodeIdInfo > (see http://wiki.apache.org/cassandra/MemtableSSTable) > DEBUG 22:50:29,147 Removing compacted SSTable files from Versions (see > http://wiki.apache.org/cassandra/MemtableSSTable) > DEBUG 22:50:29,149 Removing compacted SSTable files from > Versions.76657273696f6e (see > http://wiki.apache.org/cassandra/MemtableSSTable) > DEBUG 22:50:29,150 Removing compacted SSTable files from IndexInfo > (see http://wiki.apache.org/cassandra/MemtableSSTable) > DEBUG 22:50:29,151 Removing compacted SSTable files from Schema (see > http://wiki.apache.org/cassandra/MemtableSSTable) > DEBUG 22:50:29,153 Removing compacted SSTable files from Migrations > (see http://wiki.apache.org/cassandra/MemtableSSTable) > DEBUG 22:50:29,155 Removing compacted SSTable files from LocationInfo > (see http://wiki.apache.org/cassandra/MemtableSSTable) > DEBUG 22:50:29,157 Removing compacted SSTable files from > HintsColumnFamily (see > http://wiki.apache.org/cassandra/MemtableSSTable) > DEBUG 22:50:29,158 Removing compacted SSTable files from Temps (see > http://wiki.apache.org/cassandra/MemtableSSTable) > DEBUG 22:50:29,160 Removing compacted SSTable files from > TICStatusCurrent (see > http://wiki.apache.org/cassandra/MemtableSSTable) > DEBUG 22:50:29,161 opening keyspace system > DEBUG 22:50:29,161 opening keyspace Innovari > DEBUG 22:50:29,162 Configured datacenter replicas are datacenter1:1 > DEBUG 22:50:29,162 Initializing Innovari.Temps > DEBUG 22:50:29,164 Starting CFS Temps > INFO 22:50:29,166 Opening > /var/lib/cassandra/data/Innovari/Temps-g-117 (156 bytes) > DEBUG 22:50:29,166 Load metadata for > /var/lib/cassandra/data/Innovari/Temps-g-117 > INFO 22:50:29,166 Opening > /var/lib/cassandra/data/Innovari/Temps-g-118 (156 bytes) > DEBUG 22:50:29,167 Load metadata for > /var/lib/cassandra/data/Innovari/Temps-g-118 > java.lang.OutOfMemoryError: Java heap space > Dumping heap to /var/lib/cassandra/java_1341615025.hprof ... > Heap dump file created [47294429 bytes in 0.418 secs] > ERROR 22:50:29,695 Fatal exception in thread Thread[SSTableBatchOpen:2,5,main] > java.lang.OutOfMemoryError: Java heap space > at > org.apache.cassandra.utils.EstimatedHistogram$EstimatedHistogramSerializer.deserialize(EstimatedHistogram.java:222) > at > org.apache.cassandra.io.sstable.SSTableMetadata$SSTableMetadataSerializer.deserialize(SSTableMetadata.java:204) > at > org.apache.cassandra.io.sstable.SSTableMetadata$SSTableMetadataSerializer.deserialize(SSTableMetadata.java:194) > at > org.apache.cassandra.io.sstable.SSTableReader.open(SSTableReader.java:155) > at > org.apache.cassandra.io.sstable.SSTableReader$1.run(SSTableReader.java:224) > at > java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:441) > at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303) > at java.util.concurrent.FutureTask.run(FutureTask.java:138) > at > java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) > at java.lang.Thread.run(Thread.java:662) > Service exit with a return value of 100 -- Jonathan Ellis Project Chair, Apache Cassandra co-founder of DataStax, the source for professional Cassandra support http://www.datastax.com
