Ryen created HDDS-15916:
---------------------------
Summary: Native memory leak in ECKeyOutputStream:
encoder.release() not called in close(), causing ISA-L native memory
accumulation
Key: HDDS-15916
URL: https://issues.apache.org/jira/browse/HDDS-15916
Project: Apache Ozone
Issue Type: Bug
Components: s3gateway
Affects Versions: 2.1.0
Environment: OS: CentOS 7 (Linux 3.10 kernel)
JDK: OpenJDK 17.0.17+10-LTS
Ozone: 2.1.0
Hadoop: 3.4.2
Ratis: 3.2.0
ISA-L: 2.32.0
JVM flags: -XX:+UseG1GC -Xmx8g -X2 with
MALLOC_CONF=background_thread:true,dirty_decay_ms:5000,muzzy_decay_ms:5000,
MALLOC_ARENA_MAX=4
S3G uptime before leak became critical: ~26 days
Reporter: Ryen
The S3 Gateway (S3G) process leaks native memory continuously when handling PUT
requests to EC (Erasure Coded) buckets. The leak is caused by
\{{ECKeyOutputStream.close()}} not invoking \{{encoder.release()}}, leaving the
underlying ISA-L native encoder context (allocated via JNI inside Hadoop's
\{{NativeRSRawEncoder}}) un-released.
h3. Symptoms
After ~26 days of S3G uptime serving a workload that includes EC-bucket PUTs:
||Metric||Value|| |JVM Xmx|8 GB| |JVM Xms|4 GB| |Heap used|~2.5 GB (\{{jmap
--heap}})| |DirectByteBuffer (Java NIO Bits)|0 (\{{jshell}} BufferPoolMXBean)|
|MappedByteBuffer|0| |Process RSS|~57 GB (\{{ps}})| |Number of large anonymous
mmap segments (>5 GB RSS each)|5| |Total RSS in those 5 segments|~33 GB|
The 5 large \{{[anon]}} segments (virtual size 6-11 GB each, RSS 5-8 GB each)
are all \{{Private_Dirty}} with only a few KB of \{{LazyFree}}, confirmed by
{{/proc/<pid>/smaps}}. \{{GC.run}} via \{{jcmd}} does not reduce RSS.
h3. Root cause
{\{ECKeyOutputStream}} creates a \{{RawErasureEncoder}} in its constructor.
When the EC replication config uses \{{rs}} codec and ISA-L native is
available, this resolves to \{{NativeRSRawEncoder}}, which wraps Hadoop's
\{{org.apache.hadoop.io.erasurecode.rawcoder.NativeRSRawEncoder}}. The Hadoop
encoder allocates native memory (ISA-L encoder context + matrix + scratch
buffers) via JNI on construction, and that memory is only freed when
\{{release()}} is called.
However, \{{ECKeyOutputStream.close()}} never calls \{{encoder.release()}}:
{code:java} //
hadoop-ozone/client/src/main/java/org/apache/hadoop/ozone/client/io/ECKeyOutputStream.java
public final class ECKeyOutputStream extends KeyOutputStream implements
KeyMetadataAware {
private final RawErasureEncoder encoder; // line 69
private ECKeyOutputStream(Builder builder) \{ ... this.encoder =
CodecUtil.createRawEncoderWithFallback( builder.getReplicationConfig()); //
line 112 ... }
@Override public void close() throws IOException \{ // line 459 if (closed) {
return; } closed = true; ... try \{ ... } catch (...) \{ ... } finally \{
closeCurrentStreamEntry(); blockOutputStreamEntryPool.cleanup(); // <--
encoder.release() is missing here } } } \{code}
The \{{release()}} method exists on \{{NativeRSRawEncoder}} (and on the Java
fallback \{{RSRawEncoder}}), but is never invoked:
{code:java} //
hadoop-hdds/erasurecode/src/main/java/org/apache/ozone/erasurecode/rawcoder/NativeRSRawEncoder.java
public class NativeRSRawEncoder extends AbstractNativeRawEncoder { private
org.apache.hadoop.io.erasurecode.rawcoder.NativeRSRawEncoder
hadoopNativeRSRawEncoder;
@Override public void release() \{ // exists, but never called
hadoopNativeRSRawEncoder.release(); } } \{code}
Every PUT to an EC bucket therefore leaks one ISA-L encoder context. Over weeks
of S3G operation this accumulates into the tens-of-GB range observed above.
h3. Why DirectByteBuffer counters show 0
The Hadoop \{{NativeRSRawEncoder}} allocates its scratch buffers via JNI
(\{{malloc}}/\{{mmap}}) rather than via \{{java.nio.Bits}}, so the leak is
invisible to \{{java.lang.management.BufferPoolMXBean}} (which reports
\{{direct = 0}} and \{{mapped = 0}}). The leaked memory shows up only as
anonymous \{{[anon]}} segments in \{{pmap}} / {{/proc/<pid>/smaps}}.
h3. Why GC.run does not help
The Hadoop native encoder does not register a \{{Cleaner}} /
\{{PhantomReference}} for its native allocation, so GC cannot reclaim it. Only
an explicit \{{release()}} call frees the native memory.
h3. Reproduction
h1. Start an Ozone cluster with S3G enabled and ISA-L native available.
h1. Create an EC bucket, e.g. \{{ozone sh bucket create ... --type EC --ec
rs-3-2-1024k}}.
h1. Drive sustained PUT traffic against the EC bucket via the S3 API (e.g. with
\{{s3cmd}}, \{{aws s3 cp}}, or a load generator).
h1. Monitor S3G RSS:
{code:bash} while true; do ps -o rss,pid -p $(pgrep -f proc_s3g); sleep 60;
done \{code}
h1. Observe RSS monotonically increasing well beyond \{{Xmx}}, while heap usage
stays low (G1 humongous allocations from the S3G \{{byte[]}} buffers are
reclaimed normally, only native memory grows).
h3. Diagnostic evidence (collected on the affected S3G)
* {{jhsdb jmap --heap --pid <pid>}}: heap 30% used, G1 healthy
* {\{jshell}} reading \{{BufferPoolMXBean}}: \{{direct = 0 MB, mapped = 0 MB}}
* {{jcmd <pid> GC.run}}: RSS unchanged afterwards
* {{jstack <pid>}}: 253 Java threads, no thread leak
* {{pmap -x <pid>}}: 5 anonymous \{{[anon]}} segments with RSS 5-8 GB each
(virtual 6-11 GB each)
* {{/proc/<pid>/smaps}} for the largest segment: \{noformat}
7f597d780000-7f5c3e400000 rw-p 00000000 00:00 0 Size: 11547136 kB Rss: 8100436
kB Private_Dirty: 8100376 kB LazyFree: 60 kB Anonymous: 8100436 kB \{noformat}
* {{perf record -e syscalls:sys_enter_mmap -p <pid>}}: only 4 KB / 16 KB
(thread stack) mmaps at steady state, confirming the large segments are
historical accumulation, not current allocation
* {{lsof -p <pid>}}: \{{libisal.so.2.0.32}} and \{{libhadoop.so}} are loaded
h3. Suggested fix
Add \{{encoder.release()}} to the \{{finally}} block of
\{{ECKeyOutputStream.close()}}:
{code:java} @Override public void close() throws IOException \{ if (closed) {
return; } closed = true; ... try \{ ... } catch (...) \{ ... } finally \{
closeCurrentStreamEntry(); blockOutputStreamEntryPool.cleanup(); if (encoder !=
null) { encoder.release(); // <-- fix } } } \{code}
{\{RawErasureEncoder}} already declares \{{release()}} (inherited via
\{{AbstractRawErasureCoder}}), so the same call works for both
\{{NativeRSRawEncoder}} (ISA-L) and the pure-Java \{{RSRawEncoder}} fallback.
This is the Ozone-side equivalent of HADOOP-17209, which fixed the same class
of leak in Hadoop HDFS's \{{ErasureCodingOutputStream}}.
h3. Workaround
Without a code fix, operators can:
h1. Restart S3G periodically (e.g. daily via cron) to reset RSS.
h1. Replace \{{NativeRSRawErasureCoderFactory}} /
\{{NativeXORRawErasureCoderFactory}} in
\{{META-INF/services/org.apache.ozone.erasurecode.rawcoder.RawErasureCoderFactory}}
inside {{hdds-erasurecode-<version>.jar}} with the pure-Java
\{{RSRawErasureCoderFactory}} / \{{XORRawErasureCoderFactory}}. This avoids the
JNI allocation entirely but loses ISA-L acceleration.
Neither workaround is ideal; a source-code fix is the proper resolution.
h3. References
* [HADOOP-17209|https://issues.apache.org/jira/browse/HADOOP-17209]: Erasure
Coding: Native library memory leak (Hadoop-side fix that Ozone should mirror on
\{{ECKeyOutputStream}})
* Ozone source:
\{{hadoop-ozone/client/src/main/java/org/apache/hadoop/ozone/client/io/ECKeyOutputStream.java}}
* Ozone source:
\{{hadoop-hdds/erasurecode/src/main/java/org/apache/ozone/erasurecode/rawcoder/NativeRSRawEncoder.java}}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
