[
https://issues.apache.org/jira/browse/CASSANDRA-6689?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13918822#comment-13918822
]
Benedict commented on CASSANDRA-6689:
-------------------------------------
bq. I can image the situation when we frequently switch memtables which, in
current code, starts a new thread, so I wonder how quickly we would be able to
see a slowdown affect caused by that + potentially OOM on stack space...
It only starts a new thread if there is GC work to be done, room to do it in,
and the static pool doesn't have any threads that haven't timed out. The
cleaner thread itself is started once and remains forever. If you mean you're
worried about flushing so many memtables that all have GC work possible on them
and that this would spam the collector thread pool, I guess I agree it's a
theoretical possibility, but it would have to be pretty extreme. We can spin a
thousand threads and not break a sweat, but even with tens of thousands of
memtables we'll hit our memory limit and stop being able to do any more GC
before we cause major problems. That said, we could safely impose a cap on the
pool if we wanted. Set it largeish so it isn't a bottleneck, but prevent this
kind of problem.
bq. I'm thinking maybe it would lessen internal complexity if we do (sort-of)
arena allocation inside instead of having groups and sub pools?
Probably not appreciably. Most of complexity is likely to remain, as it's
mostly about dealing with the interactions of writers with readers, and those
will be mostly the same regardless, since readers can touch anything at any
time. The allocator group does currently allow for some slight _optimisations_,
but I don't immediately see a way that merging with pool would reduce
_complexity_. I will give it some thought though. On this vein, I would like to
see thread local allocation buffers, but again this is an optimisation rather
than a complexity reducer.
bq. It's just an idea so people can just ignore it but what I meant there is
maybe it would be a better first step for us to consider making all of the
operations that we do more self-contained so we can track exactly where it
started and ends (by operation I mean all types of reads, writes), which could
potentially make allocator job easier as we would be able to container per
operation memory...
We're currently dealing with allocations that necessarily outlive the duration
of the operation, so I'm not sure how well that would apply here, but possibly
you have some specific examples? It may be that we decide to use this code for
creating the mutations from thrift/native/MS etc., though, in which case it
would be very helpful.
> Partially Off Heap Memtables
> ----------------------------
>
> Key: CASSANDRA-6689
> URL: https://issues.apache.org/jira/browse/CASSANDRA-6689
> Project: Cassandra
> Issue Type: New Feature
> Components: Core
> Reporter: Benedict
> Assignee: Benedict
> Fix For: 2.1 beta2
>
> Attachments: CASSANDRA-6689-small-changes.patch
>
>
> Move the contents of ByteBuffers off-heap for records written to a memtable.
> (See comments for details)
--
This message was sent by Atlassian JIRA
(v6.2#6252)
