[
https://issues.apache.org/jira/browse/HDDS-11234?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Wei-Chiu Chuang updated HDDS-11234:
-----------------------------------
Attachment: ozone_dn_leak.pdf
> Manage Netty native memory consumption
> --------------------------------------
>
> Key: HDDS-11234
> URL: https://issues.apache.org/jira/browse/HDDS-11234
> Project: Apache Ozone
> Issue Type: Sub-task
> Reporter: Wei-Chiu Chuang
> Priority: Major
> Attachments: ozone_dn_leak.pdf
>
>
> We observed an Ozone DataNode that used way too much memory with HBase
> LoadTest workload.
> Most of its memory allocation was non-heap, so we thought there might be a
> native memory leak.
> Used jemalloc and jeprof tools, I managed to produce the native memory
> allocation map, and it shows most of it comes from Unsafe_AllocateMemory.
> After some digging, it led me to this post
> [https://github.com/netty/netty/issues/11835] where it suggests Netty's
> internal memory management is to blame. A workaround is to disable it (Java
> property -Dio.netty.allocator.type=unpooled) and another is to reduce the
> native memory size using -Dio.netty.maxDirectMemory=<size>
>
> By default, the size is the same as JDK's maximum direct memory size
> (-XX:MaxDirectMemorySize), which is usally the same as max heap size (-Xmx).
> We should provide a best practice for users. In addition, we have Ratis
> shaded Netty and gRPC. They use different Netty properties to configure
> memory size (-Dio.netty.maxDirectMemory,
> -Dorg.apache.ratis.thirdparty.io.netty.maxDirectMemory) So in theory the
> memory consumption can go up to 3x of maximum heap size.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
