Wei-Chiu Chuang created HDDS-11234:
--------------------------------------
Summary: Manage Netty native memory consumption
Key: HDDS-11234
URL: https://issues.apache.org/jira/browse/HDDS-11234
Project: Apache Ozone
Issue Type: Sub-task
Reporter: Wei-Chiu Chuang
We observed an Ozone DataNode that used way too much memory with HBase LoadTest
workload.
Most of its memory allocation was non-heap, so we thought there might be a
native memory leak.
Used jemalloc and jeprof tools, I managed to produce the native memory
allocation map, and it shows most of it comes from Unsafe_AllocateMemory. After
some digging, it led me to this post
[https://github.com/netty/netty/issues/11835] where it suggests Netty's
internal memory management is to blame. A workaround is to disable it (Java
property -Dio.netty.allocator.type=unpooled) and another is to reduce the
native memory size using -Dio.netty.maxDirectMemory=<size>
By default, the size is the same as JDK's maximum direct memory size
(-XX:MaxDirectMemorySize), which is usally the same as max heap size (-Xmx). We
should provide a best practice for users. In addition, we have Ratis shaded
Netty and gRPC. They use different Netty properties to configure memory size
(-Dio.netty.maxDirectMemory,
-Dorg.apache.ratis.thirdparty.io.netty.maxDirectMemory) So in theory the memory
consumption can go up to 3x of maximum heap size.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
