[
https://issues.apache.org/jira/browse/HDDS-11234?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Wei-Chiu Chuang updated HDDS-11234:
-----------------------------------
Description:
We observed an Ozone DataNode that used way too much memory with HBase LoadTest
workload.
Most of its memory allocation was non-heap, so we thought there might be a
native memory leak.
Used jemalloc and jeprof tools, I managed to produce the native memory
allocation map, and it shows most of it comes from Unsafe_AllocateMemory. It
turns out it's not a memory leak (we were on the default leak detection level
'simple') After some digging, it led me to this post
[https://github.com/netty/netty/issues/11835] where it suggests Netty's
internal memory management is to blame. A workaround is to disable it (Java
property -Dio.netty.allocator.type=unpooled) and another is to reduce the
native memory size using -Dio.netty.maxDirectMemory=<size> Disabling it has
negative performance impact so I think controlling maximum memory size used by
Netty makes more sense.
By default, the size is the same as JDK's maximum direct memory size
(-XX:MaxDirectMemorySize), which is usally the same as max heap size (-Xmx). We
should provide a best practice for users. In addition, we have Ratis shaded
Netty and gRPC. They use different Netty properties to configure memory size
(-Dio.netty.maxDirectMemory,
-Dorg.apache.ratis.thirdparty.io.netty.maxDirectMemory) So in theory the memory
consumption can go up to 3x of maximum heap size.
was:
We observed an Ozone DataNode that used way too much memory with HBase LoadTest
workload.
Most of its memory allocation was non-heap, so we thought there might be a
native memory leak.
Used jemalloc and jeprof tools, I managed to produce the native memory
allocation map, and it shows most of it comes from Unsafe_AllocateMemory. After
some digging, it led me to this post
[https://github.com/netty/netty/issues/11835] where it suggests Netty's
internal memory management is to blame. A workaround is to disable it (Java
property -Dio.netty.allocator.type=unpooled) and another is to reduce the
native memory size using -Dio.netty.maxDirectMemory=<size> Disabling it has
negative performance impact so I think controlling maximum memory size used by
Netty makes more sense.
By default, the size is the same as JDK's maximum direct memory size
(-XX:MaxDirectMemorySize), which is usally the same as max heap size (-Xmx). We
should provide a best practice for users. In addition, we have Ratis shaded
Netty and gRPC. They use different Netty properties to configure memory size
(-Dio.netty.maxDirectMemory,
-Dorg.apache.ratis.thirdparty.io.netty.maxDirectMemory) So in theory the memory
consumption can go up to 3x of maximum heap size.
> Manage Netty native memory consumption
> --------------------------------------
>
> Key: HDDS-11234
> URL: https://issues.apache.org/jira/browse/HDDS-11234
> Project: Apache Ozone
> Issue Type: Sub-task
> Reporter: Wei-Chiu Chuang
> Priority: Major
> Attachments: Screenshot 2024-07-25 at 10.32.40 AM.png,
> ozone_dn_leak.pdf
>
>
> We observed an Ozone DataNode that used way too much memory with HBase
> LoadTest workload.
> Most of its memory allocation was non-heap, so we thought there might be a
> native memory leak.
> Used jemalloc and jeprof tools, I managed to produce the native memory
> allocation map, and it shows most of it comes from Unsafe_AllocateMemory. It
> turns out it's not a memory leak (we were on the default leak detection level
> 'simple') After some digging, it led me to this post
> [https://github.com/netty/netty/issues/11835] where it suggests Netty's
> internal memory management is to blame. A workaround is to disable it (Java
> property -Dio.netty.allocator.type=unpooled) and another is to reduce the
> native memory size using -Dio.netty.maxDirectMemory=<size> Disabling it has
> negative performance impact so I think controlling maximum memory size used
> by Netty makes more sense.
>
> By default, the size is the same as JDK's maximum direct memory size
> (-XX:MaxDirectMemorySize), which is usally the same as max heap size (-Xmx).
> We should provide a best practice for users. In addition, we have Ratis
> shaded Netty and gRPC. They use different Netty properties to configure
> memory size (-Dio.netty.maxDirectMemory,
> -Dorg.apache.ratis.thirdparty.io.netty.maxDirectMemory) So in theory the
> memory consumption can go up to 3x of maximum heap size.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
