Re: CMS GC / fragmentation / memtables etc

2014-06-17 Thread Benedict Elliott Smith
The discussion about virtual methods was around a number of considerations, as there was some work to refactor the hierarchy and introduce more virtual methods for a single allocator's cell hierarchies. The allocator also does not determine the total set of possible cell implementations, as

Re: CMS GC / fragmentation / memtables etc

2014-06-15 Thread graham sanderson
Hi Benedict, So I had a look at the code, and as you say it looked pretty easy to recycle on heap slabs… there is already RACE_ALLOCATED which keeps a strongly referenced pool, however I was thinking in this case of just WeakReferences. In terms of on heap slabs, it seemed to me that recycling

Re: CMS GC / fragmentation / memtables etc

2014-06-15 Thread Benedict Elliott Smith
Hi Graham, Unfortunately the problem is more difficult than you might think. References to the buffers can persist in-flight to clients long after the memtable is discarded, so you would be introducing a subtle corruption risk for data returned to clients. Unfortunately the current implementation

Re: CMS GC / fragmentation / memtables etc

2014-06-15 Thread graham sanderson
Hi Benedict, Ah, my mistake, I had assumed that since the memory backing the off heap ByteBuffer Regions was freed on discard, that the Regions would be immediately recyclable in the on heap case. I guess I’ll just configure for one of the off-heap variants when 2.1 comes out… Any idea of the

Re: CMS GC / fragmentation / memtables etc

2014-06-15 Thread Benedict Elliott Smith
The current implementation is slower for a single memtable-only read (dependent on workload, but similar ball park to on heap), but can store substantially more per memtable under offheap_objects (again dependent on workload, but as much as 8x in extreme cases) which means more queries are

Re: CMS GC / fragmentation / memtables etc

2014-06-15 Thread graham sanderson
Hi Benedict thanks once again for your explanations (hopefully these are my last questions!)… I just read https://issues.apache.org/jira/browse/CASSANDRA-6694 really quickly, so didn’t follow all the discussions…. one thing to note that was argued in the middle was about the cost of virtual

Re: CMS GC / fragmentation / memtables etc

2014-05-21 Thread Benedict Elliott Smith
Graham, This is largely fixed in 2.1 with the introduction of partially off-heap memtables - the slabs reside off-heap, so do not cause any GC issues. As it happens the changes would also permit us to recycle on-heap slabs reasonable easily as well, so feel free to file a ticket for that,

Re: CMS GC / fragmentation / memtables etc

2014-05-21 Thread graham sanderson
Thanks so much for the info… I will read the relevant code and/or JIRA issues since I’m not sure what the trigger for on/off heap is yet, and maybe open a ticket for recycling. Hopefully in the short term we can mitigate our fragmentation problem trivially - we upped JVM memory on some nodes

CMS GC / fragmentation / memtables etc

2014-05-20 Thread graham sanderson
So i’ve been tinkering a bit with CMS config because we are still seeing fairly frequent full compacting GC due to framgentation/promotion failure As mentioned below, we are usually too fragmented to promote new in-flight memtables. This is likely caused by sudden write spikes (which we do