Hi Stefania, Thanks for the explanation - I think it may be more likely I'm hitting MEMORY_USAGE_THRESHOLD (file_cache_size_in_mb, which is 512mb by default), as the table itself is very large (a couple TB over 12 nodes). In that case, due to the changes outlined CASSANDRA-9548/CASSANDRA-8709/CASSANDRA-5863, there should probably be a warning in 3.6 that buffer_pool_use_heap_if_exhausted should be set to false if you still have some tables on SnappyCompression.
Nimi On Wed, Jun 22, 2016 at 12:14 AM, Stefania Alborghetti < stefania.alborghe...@datastax.com> wrote: > Hi Nimi > > I am not familiar with how CASSANDRA-5863 uses the buffer pool yet, but I > do know how the buffer pool works > > It has a local pool, which is thread local, and takes chunks from a global > pool. These are allocated off heap and aligned to the page cache > boundaries, see here > < > https://github.com/apache/cassandra/blob/30bb255ec9fb36ace2aab51474bd3bfb9bbd3bed/src/java/org/apache/cassandra/utils/memory/BufferPool.java#L276 > >. > > > The buffer pool will allocate according to > buffer_pool_use_heap_if_exhausted > only in two cases: > > - The global pool has run out of chunks, which happens when there is an > OOM exception or when MEMORY_USAGE_THRESHOLD is exhausted, see here > < > https://github.com/apache/cassandra/blob/30bb255ec9fb36ace2aab51474bd3bfb9bbd3bed/src/java/org/apache/cassandra/utils/memory/BufferPool.java#L259 > > > . > - The requested buffer size is larger than a chunk, currently > CHUNK_SIZE, 64 kb. > > To understand this behaviour look at maybeTakeFromPool > < > https://github.com/apache/cassandra/blob/30bb255ec9fb36ace2aab51474bd3bfb9bbd3bed/src/java/org/apache/cassandra/utils/memory/BufferPool.java#L124 > >(), > the line that takes from the local pool is the last one and follow from > there. > > The original ticket for the buffer pool is CASSANDRA-8897 > <https://issues.apache.org/jira/browse/CASSANDRA-8897>if you need more > details, and there is CASSANDRA-9468 > <https://issues.apache.org/jira/browse/CASSANDRA-9468> to extend it to > allow chunks with bigger or smaller sizes (point 4). > > I've left a comment on CASSANDRA-11993. > > I hope this helps, > Stefania > > > On Wed, Jun 22, 2016 at 4:54 AM, Nimi Wariboko Jr <n...@channelmeter.com> > wrote: > > > Hi, > > > > I'm trying to debug an issue I'm having with Cassandra 3.6+ and Snappy > > compressed tables - CASSANDRA-11993 (that may be related to > > CASSANDRA-5863). > > > > I'm probably 3% familiar with Cassandra internals, so apologies if I'm > > wrong here - I was looking at a change in CASSANDRA-5863 where a cache > was > > added to the SSTableIterator code path, which may have > > included org.apache.cassandra.utils.memory.BufferPool into the read path. > > > > My confusion comes from here - AFAICT, when the BufferPool needs to > > allocate more memory, it calls allocate(int size, boolean onHeap) ( > > > > > https://github.com/apache/cassandra/blob/30bb255ec9fb36ace2aab51474bd3bfb9bbd3bed/src/java/org/apache/cassandra/utils/memory/BufferPool.java#L82 > > ). > > The value for whether onHeap is true is > > constant ALLOCATE_ON_HEAP_WHEN_EXAHUSTED. > > > > ALLOCATE_ON_HEAP_WHEN_EXAHUSTED is defined by the yaml > > setting buffer_pool_use_heap_if_exhausted which is true by default (and > was > > so in my install). > > > > Further still ( > > > > > https://github.com/apache/cassandra/blob/30bb255ec9fb36ace2aab51474bd3bfb9bbd3bed/src/java/org/apache/cassandra/utils/memory/BufferPool.java > > ), > > I can't see any logic where BufferPool will actually allocate off heap > > memory - it seems, due to the buffer_pool_use_heap_if_exhausted setting, > > always allocate memory on heap. It never checks if the off heap memory is > > exhausted (AFAICT from these 3 files). > > > > Sure enough, if I set buffer_pool_use_heap_if_exhausted, my issue at > > CASSANDRA-11993 goes away (I think? I'm having a number of issues lately > > and I may have traded one exception for another). > > > > Is this the correct behavior of buffer_pool_use_heap_if_exhausted? > > > > Nimi > > > > > > -- > > > [image: datastax_logo.png] <http://www.datastax.com/> > > Stefania Alborghetti > > Apache Cassandra Software Engineer > > |+852 6114 9265| stefania.alborghe...@datastax.com > > > [image: cassandrasummit.org/Email_Signature] > <http://cassandrasummit.org/Email_Signature> >
