[
https://issues.apache.org/jira/browse/HDDS-2342?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16957212#comment-16957212
]
Attila Doroszlai commented on HDDS-2342:
----------------------------------------
Maximum number of entries in {{stateMachineDataCache}} is also based on chunk
executor thread count, so this also affects heap size requirement.
> ContainerStateMachine$chunkExecutor threads hold onto native memory
> -------------------------------------------------------------------
>
> Key: HDDS-2342
> URL: https://issues.apache.org/jira/browse/HDDS-2342
> Project: Hadoop Distributed Data Store
> Issue Type: Bug
> Components: Ozone Datanode
> Reporter: Lokesh Jain
> Assignee: Lokesh Jain
> Priority: Major
>
> In a heap dump many threads in ContainerStateMachine$chunkExecutor holds onto
> native memory in the ThreadLocal map. Every such thread holds onto chunk
> worth of DirectByteBuffer. Since these threads are involved in write and read
> chunk operations, the JVM allocates chunk (16MB) worth of DirectByteBuffer in
> the ThreadLocalMap for every thread involved in IO. Also the native memory
> would not be GC'ed as long as the thread is alive.
> It would be better to reduce the default number of chunk executor threads and
> have them in proportion to number of disks on the datanode. We should also
> use DirectByeBuffers for the IO on datanode. Currently we allocate
> HeapByteBuffer which needs to be backed by DirectByteBuffer. If we can use a
> DirectByteBuffer we can avoid a buffer copy.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
---------------------------------------------------------------------
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
