[
https://issues.apache.org/jira/browse/CASSANDRA-11870?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15344121#comment-15344121
]
Robert Stupp commented on CASSANDRA-11870:
------------------------------------------
It's a bit difficult to tell whether it's experienced in the wild, since before
3.x Netty buffers were on-heap and C* only had somewhat infrequent direct
memory allocations. However, with 3.0 and especially 3.6, we have more frequent
off-heap memory allocations. At least from my point of view, the majority is
still on 2.1.
What I intend to tackle is to prevent the (ugly) call to
{{java.nio.Bits#reserveMemory}} and also the global linked queue via
{{sun.misc.Cleaner}}, which has static synchronized methods {{add}} +
{{remove}}. My objection about {{ByteBuffer.allocateDirect()}} in general is
that if one thread runs into an _out-of-direct-memory_ situations, other thread
will likely run into that situation, too - and that will cause one CMS-GC
({{System.gc()}}) per thread.
However, I agree that maintaining a separate {{max_direct_memory_in_mb}} config
variable is probably not the best way. A better option could be to use the
already existing options that define off-heap memory bounds like
{{file_cache_size_in_mb}}, {{memtable_offheap_space_in_mb}},
{{index_summary_capacity_in_mb}}, sum them and use that value. WDYT?
I'm not sure, whether the following is a good idea: while testing OHC a while
ago with >30G of small allocations, it worked really nice (OHC does not use
BB.allocateDirect - that time I just realized that it's slow). What I'm
thinking of, is to not pool off-heap memory at all (and just keep track of the
total allocated amount, as currently in the patch) and leave alloc()/free() up
to the OS or jemalloc.
Your idea to centralize BB allocations in BufferPool sounds good!
> Consider allocating direct buffers bypassing ByteBuffer.allocateDirect
> ----------------------------------------------------------------------
>
> Key: CASSANDRA-11870
> URL: https://issues.apache.org/jira/browse/CASSANDRA-11870
> Project: Cassandra
> Issue Type: Improvement
> Reporter: Robert Stupp
> Assignee: Robert Stupp
> Priority: Minor
> Fix For: 3.x
>
>
> As outlined in CASSANDRA-11818, {{ByteBuffer.allocateDirect}} uses
> {{Bits.reserveMemory}}, which is there to respect the JVM setting
> {{-XX:MaxDirectMemorySize=...}}.
> {{Bits.reserveMemory}} first tries an "optimistic" {{tryReserveMemory}} and
> exits immediately on success. However, if that somehow doesn't succeed, it
> triggers a {{System.gc()}}, which is bad IMO (however, kind of how direct
> buffers work in Java). After that GC it sleeps and tries to reserve the
> memory up to 9 times - up to 511 ms - and then throws
> {{OutOfMemoryError("Direct buffer memory")}}.
> This is unnecessary for us since we always immediately "free" direct buffers
> as soon as we no longer need them.
> Proposal: Manage direct-memory reservations in our own code and skip
> {{Bits.reserveMemory}} that way.
> (However, Netty direct buffers are not under our control.)
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
