Re: OOM while reading key cache
Yes, as I wrote in first e-mail. When I removed key cache file cassandra started without further problems. regards Olek 2013/11/13 Robert Coli rc...@eventbrite.com: On Wed, Nov 13, 2013 at 12:35 AM, Tom van den Berge t...@drillster.com wrote: I'm having the same problem, after upgrading from 1.2.3 to 1.2.10. I can remember this was a bug that was solved in the 1.0 or 1.1 version some time ago, but apparently it got back. A workaround is to delete the contents of the saved_caches directory before starting up. Yours is not the first report of this I've heard resulting from a 1.2.x to 1.2.x upgrade. Reports are of the form I had to nuke my saved_caches or I couldn't start my node, it OOMED, etc.. https://issues.apache.org/jira/browse/CASSANDRA-6325 Exists, but doesn't seem to be the same issue. https://issues.apache.org/jira/browse/CASSANDRA-5986 Similar, doesn't seem to be an issue triggered by upgrade.. If I were one of the posters on this thread, I would strongly consider filing a JIRA on point. @OP (olek) : did removing the saved_caches also fix your problem? =Rob
Re: OOM while reading key cache
A few month ago, we've got a similar issue on 1.2.6 : https://issues.apache.org/jira/browse/CASSANDRA-5706 But it has been fixed and did not encountered this issue anymore (we're also on 1.2.10) 2013/11/14 olek.stas...@gmail.com olek.stas...@gmail.com Yes, as I wrote in first e-mail. When I removed key cache file cassandra started without further problems. regards Olek 2013/11/13 Robert Coli rc...@eventbrite.com: On Wed, Nov 13, 2013 at 12:35 AM, Tom van den Berge t...@drillster.com wrote: I'm having the same problem, after upgrading from 1.2.3 to 1.2.10. I can remember this was a bug that was solved in the 1.0 or 1.1 version some time ago, but apparently it got back. A workaround is to delete the contents of the saved_caches directory before starting up. Yours is not the first report of this I've heard resulting from a 1.2.x to 1.2.x upgrade. Reports are of the form I had to nuke my saved_caches or I couldn't start my node, it OOMED, etc.. https://issues.apache.org/jira/browse/CASSANDRA-6325 Exists, but doesn't seem to be the same issue. https://issues.apache.org/jira/browse/CASSANDRA-5986 Similar, doesn't seem to be an issue triggered by upgrade.. If I were one of the posters on this thread, I would strongly consider filing a JIRA on point. @OP (olek) : did removing the saved_caches also fix your problem? =Rob -- Fabien Rousseau aur...@yakaz.comwww.yakaz.com
Re: OOM while reading key cache
On Wed, Nov 13, 2013 at 12:35 AM, Tom van den Berge t...@drillster.comwrote: I'm having the same problem, after upgrading from 1.2.3 to 1.2.10. I can remember this was a bug that was solved in the 1.0 or 1.1 version some time ago, but apparently it got back. A workaround is to delete the contents of the saved_caches directory before starting up. Yours is not the first report of this I've heard resulting from a 1.2.x to 1.2.x upgrade. Reports are of the form I had to nuke my saved_caches or I couldn't start my node, it OOMED, etc.. https://issues.apache.org/jira/browse/CASSANDRA-6325 Exists, but doesn't seem to be the same issue. https://issues.apache.org/jira/browse/CASSANDRA-5986 Similar, doesn't seem to be an issue triggered by upgrade.. If I were one of the posters on this thread, I would strongly consider filing a JIRA on point. @OP (olek) : did removing the saved_caches also fix your problem? =Rob
Re: OOM while reading key cache
-6 machines which 8gb RAM each and three 150GB disks each -default heap configuration With 8GB the default heap is 2GB, try kicking that up to 4GB and a 600 to 800 MB new heap. I would guess for the data load you have 2GB is not enough. hope that helps. - Aaron Morton New Zealand @aaronmorton Co-Founder Principal Consultant Apache Cassandra Consulting http://www.thelastpickle.com On 8/11/2013, at 11:31 pm, olek.stas...@gmail.com wrote: Hello, I'm facing OOM on reading key_cache Cluster conf is as follows: -6 machines which 8gb RAM each and three 150GB disks each -default heap configuration -deafult key cache configuration -the biggest keyspace has abt 500GB size (RF: 2, so in fact there is 250GB of raw data). After upgrading first of the machines from 1.2.11 to 2.0.2 i've recieved error: INFO [main] 2013-11-08 10:53:16,716 AutoSavingCache.java (line 114) reading saved cache /home/synat/nosql_filesystem/cassandra/data/saved_caches/production_storage-METADATA-KeyCache-b.db ERROR [main] 2013-11-08 10:53:16,895 CassandraDaemon.java (line 478) Exception encountered during startup java.lang.OutOfMemoryError: Java heap space at org.apache.cassandra.utils.ByteBufferUtil.read(ByteBufferUtil.java:394) at org.apache.cassandra.utils.ByteBufferUtil.readWithLength(ByteBufferUtil.java:355) at org.apache.cassandra.service.CacheService$KeyCacheSerializer.deserialize(CacheService.java:352) at org.apache.cassandra.cache.AutoSavingCache.loadSaved(AutoSavingCache.java:119) at org.apache.cassandra.db.ColumnFamilyStore.init(ColumnFamilyStore.java:264) at org.apache.cassandra.db.ColumnFamilyStore.createColumnFamilyStore(ColumnFamilyStore.java:409) at org.apache.cassandra.db.ColumnFamilyStore.createColumnFamilyStore(ColumnFamilyStore.java:381) at org.apache.cassandra.db.Keyspace.initCf(Keyspace.java:314) at org.apache.cassandra.db.Keyspace.init(Keyspace.java:268) at org.apache.cassandra.db.Keyspace.open(Keyspace.java:110) at org.apache.cassandra.db.Keyspace.open(Keyspace.java:88) at org.apache.cassandra.service.CassandraDaemon.setup(CassandraDaemon.java:274) at org.apache.cassandra.service.CassandraDaemon.activate(CassandraDaemon.java:461) at org.apache.cassandra.service.CassandraDaemon.main(CassandraDaemon.java:504) Error appears every start, so I've decided to disable key cache (this was not helpful) and temporarily moved key cache out of cache folder (file was of size 13M). That helps in starting node, but this is only workaround and it's not demanded configuration. Anyone has any idea what is the real cause of problem with oom? best regards Aleksander ps. I've still 5 nodes to upgrade, I'll inform if problem apperas on the rest.
OOM while reading key cache
Hello, I'm facing OOM on reading key_cache Cluster conf is as follows: -6 machines which 8gb RAM each and three 150GB disks each -default heap configuration -deafult key cache configuration -the biggest keyspace has abt 500GB size (RF: 2, so in fact there is 250GB of raw data). After upgrading first of the machines from 1.2.11 to 2.0.2 i've recieved error: INFO [main] 2013-11-08 10:53:16,716 AutoSavingCache.java (line 114) reading saved cache /home/synat/nosql_filesystem/cassandra/data/saved_caches/production_storage-METADATA-KeyCache-b.db ERROR [main] 2013-11-08 10:53:16,895 CassandraDaemon.java (line 478) Exception encountered during startup java.lang.OutOfMemoryError: Java heap space at org.apache.cassandra.utils.ByteBufferUtil.read(ByteBufferUtil.java:394) at org.apache.cassandra.utils.ByteBufferUtil.readWithLength(ByteBufferUtil.java:355) at org.apache.cassandra.service.CacheService$KeyCacheSerializer.deserialize(CacheService.java:352) at org.apache.cassandra.cache.AutoSavingCache.loadSaved(AutoSavingCache.java:119) at org.apache.cassandra.db.ColumnFamilyStore.init(ColumnFamilyStore.java:264) at org.apache.cassandra.db.ColumnFamilyStore.createColumnFamilyStore(ColumnFamilyStore.java:409) at org.apache.cassandra.db.ColumnFamilyStore.createColumnFamilyStore(ColumnFamilyStore.java:381) at org.apache.cassandra.db.Keyspace.initCf(Keyspace.java:314) at org.apache.cassandra.db.Keyspace.init(Keyspace.java:268) at org.apache.cassandra.db.Keyspace.open(Keyspace.java:110) at org.apache.cassandra.db.Keyspace.open(Keyspace.java:88) at org.apache.cassandra.service.CassandraDaemon.setup(CassandraDaemon.java:274) at org.apache.cassandra.service.CassandraDaemon.activate(CassandraDaemon.java:461) at org.apache.cassandra.service.CassandraDaemon.main(CassandraDaemon.java:504) Error appears every start, so I've decided to disable key cache (this was not helpful) and temporarily moved key cache out of cache folder (file was of size 13M). That helps in starting node, but this is only workaround and it's not demanded configuration. Anyone has any idea what is the real cause of problem with oom? best regards Aleksander ps. I've still 5 nodes to upgrade, I'll inform if problem apperas on the rest.
