[
https://issues.apache.org/jira/browse/HDDS-2342?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Lokesh Jain updated HDDS-2342:
------------------------------
Description:
In a heap dump many threads in ContainerStateMachine$chunkExecutor holds onto
native memory in the ThreadLocal map. Every such thread holds onto chunk worth
of DirectByteBuffer. Since these threads are involved in write and read chunk
operations, the JVM allocates chunk (16MB) worth of DirectByteBuffer in the
ThreadLocalMap for every thread involved in IO. Also the native memory would
not be GC'ed as long as the thread is alive.
It would be better to reduce the default number of chunk executor threads and
have them in proportion to number of disks on the datanode. We should also use
DirectByeBuffers for the IO on datanode. Currently we allocate HeapByteBuffer
which needs to be backed by DirectByteBuffer. If we can use a DirectByteBuffer
we can avoid a buffer copy.
was:
In a heap dump many threads in ContainerStateMachine$chunkExecutor holds onto
native memory in the ThreadLocal map. Every such thread holds onto chunk worth
of DirectByteBuffer. Since these threads are involved in write and read chunk
operations, the JVM allocates chunk (16MB) worth of DirectByteBuffer in the
ThreadLocalMap for every thread involved in IO. Also the native memory would
not be GC'ed as long as the thread is alive.
It would be better to reduce the default number of chunk executor threads and
have them in proportion to number of disks on the datanode.
> ContainerStateMachine$chunkExecutor threads hold onto native memory
> -------------------------------------------------------------------
>
> Key: HDDS-2342
> URL: https://issues.apache.org/jira/browse/HDDS-2342
> Project: Hadoop Distributed Data Store
> Issue Type: Bug
> Components: Ozone Datanode
> Reporter: Lokesh Jain
> Assignee: Lokesh Jain
> Priority: Major
>
> In a heap dump many threads in ContainerStateMachine$chunkExecutor holds onto
> native memory in the ThreadLocal map. Every such thread holds onto chunk
> worth of DirectByteBuffer. Since these threads are involved in write and read
> chunk operations, the JVM allocates chunk (16MB) worth of DirectByteBuffer in
> the ThreadLocalMap for every thread involved in IO. Also the native memory
> would not be GC'ed as long as the thread is alive.
> It would be better to reduce the default number of chunk executor threads and
> have them in proportion to number of disks on the datanode. We should also
> use DirectByeBuffers for the IO on datanode. Currently we allocate
> HeapByteBuffer which needs to be backed by DirectByteBuffer. If we can use a
> DirectByteBuffer we can avoid a buffer copy.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
---------------------------------------------------------------------
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
