To start up your node again, you could delete the stored key caches ( /var/lib/cassandra/saved_caches/*).
Regards, Tom On Wed, Dec 4, 2013 at 7:46 PM, Krishna Chaitanya <bnsk1990r...@gmail.com>wrote: > Hey Nate, > Thanks for the reply. The link was really good...!!! Looking > forward to making the necessary changes and trying this approach. > > Thanks. > > Regards, > BNSK. > > > On Wed, Dec 4, 2013 at 9:00 AM, Nate McCall <n...@thelastpickle.com>wrote: > >> For a limited memory environment, take a look at the following: >> >> http://www.opensourceconnections.com/2013/08/31/building-the-perfect-cassandra-test-environment/ >> >> >> >> >> On Wed, Dec 4, 2013 at 11:05 AM, Krishna Chaitanya < >> bnsk1990r...@gmail.com> wrote: >> >>> Hello, >>> I am currently using Cassandra-2.0.0 on OpenSuse for storing >>> netflow packets that are seen on my ethernet interface. I deliberately >>> tried to test Cassandra with heavy data and it ran fine for about 30 mins >>> after which it crashed with OutOfMemory error. I set up a two-node cluster >>> to which this data is getting stored with replication_factor 1. Now, >>> Cassandra is not even starting up. The log is given below for your >>> reference. >>> Can I solve >>> this problem by tweaking JVM OPTS? If yes, which all and how? How can I be >>> sure that it is not someother issue like corrupted commit log headers, etc. >>> so as to prevent these errors in the future? I am on a 32-bit OpenSuse i5 >>> machine with 4G RAM. >>> >>> Here is the output when I try to start Cassandra:- >>> >>> linux-0cpn:~/bnsk/ >>> experimentation/apache-cassandra-2.0.0/bin # ./cassandra -f & >>> [1] 984 >>> linux-0cpn:~/bnsk/experimentation/apache-cassandra-2.0.0/bin # >>> ./../conf/cassandra-env.sh: line 137: elseError:: command not found >>> xss = -ea -javaagent:./../lib/jamm-0.2.5.jar -XX:+UseThreadPriorities >>> -XX:ThreadPriorityPolicy=42 -Xms1024M -Xmx1024M -Xmn256M >>> -XX:+HeapDumpOnOutOfMemoryError -Xss256k >>> INFO 03:46:43,178 Logging initialized >>> INFO 03:46:43,188 32bit JVM detected. It is recommended to run >>> Cassandra on a 64bit JVM for better performance. >>> INFO 03:46:43,189 JVM vendor/version: Java HotSpot(TM) Server >>> VM/1.7.0_45 >>> INFO 03:46:43,189 Heap size: 1046937600/1046937600 >>> INFO 03:46:43,189 Classpath: >>> ./../conf:./../build/classes/main:./../build/classes/thrift:./../lib/antlr-3.2.jar:./../lib/apache-cassandra-2.0.0.jar:./../lib/apache-cassandra-clientutil-2.0.0.jar:./../lib/apache-cassandra-thrift-2.0.0.jar:./../lib/commons-cli-1.1.jar:./../lib/commons-codec-1.2.jar:./../lib/commons-lang-2.6.jar:./../lib/compress-lzf-0.8.4.jar:./../lib/concurrentlinkedhashmap-lru-1.3.jar:./../lib/disruptor-3.0.1.jar:./../lib/guava-13.0.1.jar:./../lib/high-scale-lib-1.1.2.jar:./../lib/jackson-core-asl-1.9.2.jar:./../lib/jackson-mapper-asl-1.9.2.jar:./../lib/jamm-0.2.5.jar:./../lib/jbcrypt-0.3m.jar:./../lib/jline-1.0.jar:./../lib/json-simple-1.1.jar:./../lib/libthrift-0.9.0.jar:./../lib/log4j-1.2.16.jar:./../lib/lz4-1.1.0.jar:./../lib/metrics-core-2.0.3.jar:./../lib/netty-3.5.9.Final.jar:./../lib/servlet-api-2.5-20081211.jar:./../lib/slf4j-api-1.7.2.jar:./../lib/slf4j-log4j12-1.7.2.jar:./../lib/snakeyaml-1.11.jar:./../lib/snappy-java-1.0.5.jar:./../lib/snaptree-0.1.jar:./../lib/thrift-server-0.3.0.jar:./../lib/jamm-0.2.5.jar >>> INFO 03:46:43,191 JNA not found. Native methods will be disabled. >>> INFO 03:46:43,199 Loading settings from >>> file:/root/bnsk/experimentation/apache-cassandra-2.0.0/conf/cassandra.yaml >>> INFO 03:46:43,418 Data files directories: [/var/lib/cassandra/data] >>> INFO 03:46:43,418 Commit log directory: /var/lib/cassandra/commitlog >>> INFO 03:46:43,418 DiskAccessMode 'auto' determined to be standard, >>> indexAccessMode is standard >>> INFO 03:46:43,418 disk_failure_policy is stop >>> INFO 03:46:43,422 Global memtable threshold is enabled at 332MB >>> INFO 03:46:43,510 Not using multi-threaded compaction >>> INFO 03:46:43,660 Initializing key cache with capacity of 49 MBs. >>> INFO 03:46:43,667 Scheduling key cache save to each 14400 seconds >>> (going to save all keys). >>> INFO 03:46:43,668 Initializing row cache with capacity of 0 MBs >>> INFO 03:46:43,674 Scheduling row cache save to each 0 seconds (going to >>> save all keys). >>> INFO 03:46:43,755 Initializing system.schema_triggers >>> INFO 03:46:43,768 Opening >>> /var/lib/cassandra/data/system/schema_triggers/system-schema_triggers-ja-104 >>> (57 bytes) >>> INFO 03:46:43,768 Opening >>> /var/lib/cassandra/data/system/schema_triggers/system-schema_triggers-ja-105 >>> (57 bytes) >>> INFO 03:46:43,769 Opening >>> /var/lib/cassandra/data/system/schema_triggers/system-schema_triggers-ja-103 >>> (57 bytes) >>> INFO 03:46:43,790 reading saved cache >>> /var/lib/cassandra/saved_caches/system-schema_triggers-KeyCache-b.db >>> INFO 03:46:43,798 Initializing system.batchlog >>> INFO 03:46:43,800 Initializing system.peer_events >>> INFO 03:46:43,804 Initializing system.compactions_in_progress >>> INFO 03:46:43,805 Opening >>> /var/lib/cassandra/data/system/compactions_in_progress/system-compactions_in_progress-ja-22 >>> (177 bytes) >>> INFO 03:46:43,805 Opening >>> /var/lib/cassandra/data/system/compactions_in_progress/system-compactions_in_progress-ja-23 >>> (42 bytes) >>> INFO 03:46:43,809 Initializing system.hints >>> INFO 03:46:43,812 Initializing system.schema_keyspaces >>> INFO 03:46:43,815 Opening >>> /var/lib/cassandra/data/system/schema_keyspaces/system-schema_keyspaces-ja-493 >>> (554 bytes) >>> INFO 03:46:43,816 Opening >>> /var/lib/cassandra/data/system/schema_keyspaces/system-schema_keyspaces-ja-494 >>> (473 bytes) >>> INFO 03:46:43,818 reading saved cache >>> /var/lib/cassandra/saved_caches/system-schema_keyspaces-KeyCache-b.db >>> INFO 03:46:43,821 Initializing system.range_xfers >>> INFO 03:46:43,824 Initializing system.schema_columnfamilies >>> INFO 03:46:43,826 Opening >>> /var/lib/cassandra/data/system/schema_columnfamilies/system-schema_columnfamilies-ja-535 >>> (77 bytes) >>> INFO 03:46:43,827 Opening >>> /var/lib/cassandra/data/system/schema_columnfamilies/system-schema_columnfamilies-ja-534 >>> (75 bytes) >>> INFO 03:46:43,833 Opening >>> /var/lib/cassandra/data/system/schema_columnfamilies/system-schema_columnfamilies-ja-533 >>> (8589 bytes) >>> INFO 03:46:43,841 reading saved cache >>> /var/lib/cassandra/saved_caches/system-schema_columnfamilies-KeyCache-b.db >>> java.lang.OutOfMemoryError: Java heap space >>> Dumping heap to java_pid984.hprof ... >>> Heap dump file created [15493326 bytes in 0.068 secs] >>> ERROR 03:46:43,995 Exception encountered during startup >>> java.lang.OutOfMemoryError: Java heap space >>> at >>> org.apache.cassandra.utils.ByteBufferUtil.read(ByteBufferUtil.java:394) >>> at >>> org.apache.cassandra.utils.ByteBufferUtil.readWithLength(ByteBufferUtil.java:355) >>> at >>> org.apache.cassandra.service.CacheService$KeyCacheSerializer.deserialize(CacheService.java:337) >>> at >>> org.apache.cassandra.cache.AutoSavingCache.loadSaved(AutoSavingCache.java:147) >>> at >>> org.apache.cassandra.db.ColumnFamilyStore.<init>(ColumnFamilyStore.java:267) >>> at >>> org.apache.cassandra.db.ColumnFamilyStore.createColumnFamilyStore(ColumnFamilyStore.java:411) >>> at >>> org.apache.cassandra.db.ColumnFamilyStore.createColumnFamilyStore(ColumnFamilyStore.java:383) >>> at org.apache.cassandra.db.Keyspace.initCf(Keyspace.java:314) >>> at org.apache.cassandra.db.Keyspace.<init>(Keyspace.java:268) >>> at org.apache.cassandra.db.Keyspace.open(Keyspace.java:110) >>> at org.apache.cassandra.db.Keyspace.open(Keyspace.java:88) >>> at >>> org.apache.cassandra.db.SystemKeyspace.checkHealth(SystemKeyspace.java:447) >>> at >>> org.apache.cassandra.service.CassandraDaemon.setup(CassandraDaemon.java:227) >>> at >>> org.apache.cassandra.service.CassandraDaemon.activate(CassandraDaemon.java:443) >>> at >>> org.apache.cassandra.service.CassandraDaemon.main(CassandraDaemon.java:486) >>> java.lang.OutOfMemoryError: Java heap space >>> at >>> org.apache.cassandra.utils.ByteBufferUtil.read(ByteBufferUtil.java:394) >>> at >>> org.apache.cassandra.utils.ByteBufferUtil.readWithLength(ByteBufferUtil.java:355) >>> at >>> org.apache.cassandra.service.CacheService$KeyCacheSerializer.deserialize(CacheService.java:337) >>> at >>> org.apache.cassandra.cache.AutoSavingCache.loadSaved(AutoSavingCache.java:147) >>> at >>> org.apache.cassandra.db.ColumnFamilyStore.<init>(ColumnFamilyStore.java:267) >>> at >>> org.apache.cassandra.db.ColumnFamilyStore.createColumnFamilyStore(ColumnFamilyStore.java:411) >>> at >>> org.apache.cassandra.db.ColumnFamilyStore.createColumnFamilyStore(ColumnFamilyStore.java:383) >>> at org.apache.cassandra.db.Keyspace.initCf(Keyspace.java:314) >>> at org.apache.cassandra.db.Keyspace.<init>(Keyspace.java:268) >>> at org.apache.cassandra.db.Keyspace.open(Keyspace.java:110) >>> at org.apache.cassandra.db.Keyspace.open(Keyspace.java:88) >>> at >>> org.apache.cassandra.db.SystemKeyspace.checkHealth(SystemKeyspace.java:447) >>> at >>> org.apache.cassandra.service.CassandraDaemon.setup(CassandraDaemon.java:227) >>> at >>> org.apache.cassandra.service.CassandraDaemon.activate(CassandraDaemon.java:443) >>> at >>> org.apache.cassandra.service.CassandraDaemon.main(CassandraDaemon.java:486) >>> Exception encountered during startup: Java heap space >>> >>> >>> Thanks in advance... >>> -- >>> Regards, >>> BNSK >>> *. * >>> >> >> >> >> -- >> ----------------- >> Nate McCall >> Austin, TX >> @zznate >> >> Co-Founder & Sr. Technical Consultant >> Apache Cassandra Consulting >> http://www.thelastpickle.com >> > > > > -- > Regards, > BNSK > *. * > -- Drillster BV Middenburcht 136 3452MT Vleuten Netherlands +31 30 755 5330 Open your free account at www.drillster.com
