Tomas Salfischberger created CASSANDRA-6191: -----------------------------------------------
Summary: Memory exhaustion with large number of compressed SSTables Key: CASSANDRA-6191 URL: https://issues.apache.org/jira/browse/CASSANDRA-6191 Project: Cassandra Issue Type: Bug Components: Core Environment: OS: Debian 7.1 Java: Oracle 1.7.0_25 Cassandra: 1.2.10 Memory: 24GB Heap: 8GB Reporter: Tomas Salfischberger Not sure "bug" is the right description, because I can't say for sure that the large number of SSTables is the cause of the memory issues. I'll share my research so far: Under high read-load with a very large number of compressed SSTables (caused by the initial default 5mb sstable_size in LCS) it seems memory is exhausted, without any room for GC to fix this. It tries to GC but doesn't reclaim much. The node first hits the "emergency valves" flushing all memtables, then reducing caches. And finally logs 0.99+ heap usages and hangs with GC failure or crashes with OutOfMemoryError. I've taken a heapdump and started analysis to find out what's wrong. The memory seems to be used by the byte[] backing the HeapByteBuffer in the "compressed" field of org.apache.cassandra.io.compress.CompressedRandomAccessReader. The byte[] are generally 65536 byes in size, matching the block-size of the compression. Looking further in the heap-dump I can see that these readers are part of the pool in org.apache.cassandra.io.util.CompressedPoolingSegmentedFile. Which is linked to the "dfile" field of org.apache.cassandra.io.sstable.SSTableReader. The dump-file lists 45248 instances of CompressedRandomAccessReader. Is this intended to go this way? Is there a leak somewhere? Or should there be an alternative strategy and/or warning for cases where a node is trying to read far too many SSTables? -- This message was sent by Atlassian JIRA (v6.1#6144)