[
https://issues.apache.org/jira/browse/CASSANDRA-6689?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13918785#comment-13918785
]
Pavel Yaskevich edited comment on CASSANDRA-6689 at 3/4/14 12:17 AM:
---------------------------------------------------------------------
bq. If I understand you correctly, you mean to deal with the situation where we
want to flush memtables quick enough that the single cleaner thread may become
the bottleneck? In which case, I don't think this should ever be a problem: the
work done by the cleaner itself is very minimal unless it's got some GC to do,
in which case it is potentially non-trivial, but still unlikely to be a
bottleneck. There might be a case for handing off the GC candidate selection to
another thread, except that we expect it to certainly be quicker than any
flush, and we want to wait for it to complete before deciding if flushing is
actually worthwhile, so I'm not sure it buys us much. Maybe we could do this
and continue to check the memory state isn't worsening, and if it is just force
a flush without waiting for the GC, but it probably is too unlikely to be of
benefit to be worth the extra complexity. That is, unless you meant some other
benefit I missed?
I can image the situation when we frequently switch memtables which, in current
code, starts a new thread, so I wonder how quickly we would be able to see a
slowdown affect caused by that + potentially OOM on stack space...
bq. The pool is global; a per-cf pool would have the issue of poor sharing of
resources, though, no? I would probably like to see us move to allocators
surviving across memtable replacement, so that we could then flatten allocator
group with allocator, but I'd prefer to save that for a later date.
I'm thinking maybe it would lessen internal complexity if we do (sort-of) arena
allocation inside instead of having groups and sub pools?
bq. Not really sure what you're suggesting here, could you elaborate?
It's an idea so people can just ignore it but what I meant there is maybe it
would be a better first step for us to consider making all of the operations
that we do more self-contained so we can track exactly where it started and
ends (by operation I mean all types of reads, writes), which could potentially
make allocator job easier as we would be able to allocate memory per operation
and get rid of RefAction as operation wouldn't be able to leak references to
outside...
was (Author: xedin):
bq. If I understand you correctly, you mean to deal with the situation where we
want to flush memtables quick enough that the single cleaner thread may become
the bottleneck? In which case, I don't think this should ever be a problem: the
work done by the cleaner itself is very minimal unless it's got some GC to do,
in which case it is potentially non-trivial, but still unlikely to be a
bottleneck. There might be a case for handing off the GC candidate selection to
another thread, except that we expect it to certainly be quicker than any
flush, and we want to wait for it to complete before deciding if flushing is
actually worthwhile, so I'm not sure it buys us much. Maybe we could do this
and continue to check the memory state isn't worsening, and if it is just force
a flush without waiting for the GC, but it probably is too unlikely to be of
benefit to be worth the extra complexity. That is, unless you meant some other
benefit I missed?
I can image the situation when we frequently switch memtables which, in current
code, starts a new thread, so I wonder how quickly we would be able to see a
slowdown affect caused by that + potentially OOM on stack space...
bq. The pool is global; a per-cf pool would have the issue of poor sharing of
resources, though, no? I would probably like to see us move to allocators
surviving across memtable replacement, so that we could then flatten allocator
group with allocator, but I'd prefer to save that for a later date.
I'm thinking maybe it would lessen internal complexity if we do (sort-of) arena
allocation inside instead of having groups and sub pools?
bq. Not really sure what you're suggesting here, could you elaborate?
It's just an idea so people can just ignore it but what I meant there is maybe
it would be a better first step for us to consider making all of the operations
that we do more self-contained so we can track exactly where it started and
ends (by operation I mean all types of reads, writes), which could potentially
make allocator job easier as we would be able to container per operation
memory...
> Partially Off Heap Memtables
> ----------------------------
>
> Key: CASSANDRA-6689
> URL: https://issues.apache.org/jira/browse/CASSANDRA-6689
> Project: Cassandra
> Issue Type: New Feature
> Components: Core
> Reporter: Benedict
> Assignee: Benedict
> Fix For: 2.1 beta2
>
>
> Move the contents of ByteBuffers off-heap for records written to a memtable.
> (See comments for details)
--
This message was sent by Atlassian JIRA
(v6.2#6252)
