[
https://issues.apache.org/jira/browse/HDDS-15916?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Anurag Parvatikar reassigned HDDS-15916:
----------------------------------------
Assignee: Anurag Parvatikar
> Native memory leak in ECKeyOutputStream: encoder.release() not called in
> close(), causing ISA-L native memory accumulation
> --------------------------------------------------------------------------------------------------------------------------
>
> Key: HDDS-15916
> URL: https://issues.apache.org/jira/browse/HDDS-15916
> Project: Apache Ozone
> Issue Type: Bug
> Components: s3gateway
> Affects Versions: 2.1.0
> Environment: OS: CentOS 7 (Linux 3.10 kernel)
> JDK: OpenJDK 17.0.17+10-LTS
> Ozone: 2.1.0
> Hadoop: 3.4.2
> Ratis: 3.2.0
> ISA-L: 2.32.0
> JVM flags: -XX:+UseG1GC -Xmx8g -X2 with
> MALLOC_CONF=background_thread:true,dirty_decay_ms:5000,muzzy_decay_ms:5000,
> MALLOC_ARENA_MAX=4
> S3G uptime before leak became critical: ~26 days
> Reporter: Ryen
> Assignee: Anurag Parvatikar
> Priority: Critical
> Labels: erasure-coding,, memory-leak,, native, s3g,
>
> The S3 Gateway (S3G) process leaks native memory continuously when handling
> PUT requests to EC (Erasure Coded) buckets. The leak is caused by
> \{{ECKeyOutputStream.close()}} not invoking \{{encoder.release()}}, leaving
> the underlying ISA-L native encoder context (allocated via JNI inside
> Hadoop's \{{NativeRSRawEncoder}}) un-released.
> h3. Symptoms
> After ~26 days of S3G uptime serving a workload that includes EC-bucket PUTs:
> ||Metric||Value|| |JVM Xmx|8 GB| |JVM Xms|4 GB| |Heap used|~2.5 GB (\{{jmap
> --heap}})| |DirectByteBuffer (Java NIO Bits)|0 (\{{jshell}}
> BufferPoolMXBean)| |MappedByteBuffer|0| |Process RSS|~57 GB (\{{ps}})|
> |Number of large anonymous mmap segments (>5 GB RSS each)|5| |Total RSS in
> those 5 segments|~33 GB|
> The 5 large \{{[anon]}} segments (virtual size 6-11 GB each, RSS 5-8 GB each)
> are all \{{Private_Dirty}} with only a few KB of \{{LazyFree}}, confirmed by
> {{/proc/<pid>/smaps}}. \{{GC.run}} via \{{jcmd}} does not reduce RSS.
> h3. Root cause
> {\{ECKeyOutputStream}} creates a \{{RawErasureEncoder}} in its constructor.
> When the EC replication config uses \{{rs}} codec and ISA-L native is
> available, this resolves to \{{NativeRSRawEncoder}}, which wraps Hadoop's
> \{{org.apache.hadoop.io.erasurecode.rawcoder.NativeRSRawEncoder}}. The Hadoop
> encoder allocates native memory (ISA-L encoder context + matrix + scratch
> buffers) via JNI on construction, and that memory is only freed when
> \{{release()}} is called.
> However, \{{ECKeyOutputStream.close()}} never calls \{{encoder.release()}}:
> {code:java} //
> hadoop-ozone/client/src/main/java/org/apache/hadoop/ozone/client/io/ECKeyOutputStream.java
> public final class ECKeyOutputStream extends KeyOutputStream implements
> KeyMetadataAware {
> private final RawErasureEncoder encoder; // line 69
> private ECKeyOutputStream(Builder builder) \{ ... this.encoder =
> CodecUtil.createRawEncoderWithFallback( builder.getReplicationConfig()); //
> line 112 ... }
> @Override public void close() throws IOException \{ // line 459 if (closed) {
> return; } closed = true; ... try \{ ... } catch (...) \{ ... } finally \{
> closeCurrentStreamEntry(); blockOutputStreamEntryPool.cleanup(); // <--
> encoder.release() is missing here } } } \{code}
> The \{{release()}} method exists on \{{NativeRSRawEncoder}} (and on the Java
> fallback \{{RSRawEncoder}}), but is never invoked:
> {code:java} //
> hadoop-hdds/erasurecode/src/main/java/org/apache/ozone/erasurecode/rawcoder/NativeRSRawEncoder.java
> public class NativeRSRawEncoder extends AbstractNativeRawEncoder { private
> org.apache.hadoop.io.erasurecode.rawcoder.NativeRSRawEncoder
> hadoopNativeRSRawEncoder;
> @Override public void release() \{ // exists, but never called
> hadoopNativeRSRawEncoder.release(); } } \{code}
> Every PUT to an EC bucket therefore leaks one ISA-L encoder context. Over
> weeks of S3G operation this accumulates into the tens-of-GB range observed
> above.
> h3. Why DirectByteBuffer counters show 0
> The Hadoop \{{NativeRSRawEncoder}} allocates its scratch buffers via JNI
> (\{{malloc}}/\{{mmap}}) rather than via \{{java.nio.Bits}}, so the leak is
> invisible to \{{java.lang.management.BufferPoolMXBean}} (which reports
> \{{direct = 0}} and \{{mapped = 0}}). The leaked memory shows up only as
> anonymous \{{[anon]}} segments in \{{pmap}} / {{/proc/<pid>/smaps}}.
> h3. Why GC.run does not help
> The Hadoop native encoder does not register a \{{Cleaner}} /
> \{{PhantomReference}} for its native allocation, so GC cannot reclaim it.
> Only an explicit \{{release()}} call frees the native memory.
> h3. Reproduction
> h1. Start an Ozone cluster with S3G enabled and ISA-L native available.
> h1. Create an EC bucket, e.g. \{{ozone sh bucket create ... --type EC --ec
> rs-3-2-1024k}}.
> h1. Drive sustained PUT traffic against the EC bucket via the S3 API (e.g.
> with \{{s3cmd}}, \{{aws s3 cp}}, or a load generator).
> h1. Monitor S3G RSS:
> {code:bash} while true; do ps -o rss,pid -p $(pgrep -f proc_s3g); sleep 60;
> done \{code}
> h1. Observe RSS monotonically increasing well beyond \{{Xmx}}, while heap
> usage stays low (G1 humongous allocations from the S3G \{{byte[]}} buffers
> are reclaimed normally, only native memory grows).
> h3. Diagnostic evidence (collected on the affected S3G)
> * {{jhsdb jmap --heap --pid <pid>}}: heap 30% used, G1 healthy
> * {\{jshell}} reading \{{BufferPoolMXBean}}: \{{direct = 0 MB, mapped = 0
> MB}}
> * {{jcmd <pid> GC.run}}: RSS unchanged afterwards
> * {{jstack <pid>}}: 253 Java threads, no thread leak
> * {{pmap -x <pid>}}: 5 anonymous \{{[anon]}} segments with RSS 5-8 GB each
> (virtual 6-11 GB each)
> * {{/proc/<pid>/smaps}} for the largest segment: \{noformat}
> 7f597d780000-7f5c3e400000 rw-p 00000000 00:00 0 Size: 11547136 kB Rss:
> 8100436 kB Private_Dirty: 8100376 kB LazyFree: 60 kB Anonymous: 8100436 kB
> \{noformat}
> * {{perf record -e syscalls:sys_enter_mmap -p <pid>}}: only 4 KB / 16 KB
> (thread stack) mmaps at steady state, confirming the large segments are
> historical accumulation, not current allocation
> * {{lsof -p <pid>}}: \{{libisal.so.2.0.32}} and \{{libhadoop.so}} are loaded
> h3. Suggested fix
> Add \{{encoder.release()}} to the \{{finally}} block of
> \{{ECKeyOutputStream.close()}}:
> {code:java} @Override public void close() throws IOException \{ if (closed) {
> return; } closed = true; ... try \{ ... } catch (...) \{ ... } finally \{
> closeCurrentStreamEntry(); blockOutputStreamEntryPool.cleanup(); if (encoder
> != null) { encoder.release(); // <-- fix } } } \{code}
> {\{RawErasureEncoder}} already declares \{{release()}} (inherited via
> \{{AbstractRawErasureCoder}}), so the same call works for both
> \{{NativeRSRawEncoder}} (ISA-L) and the pure-Java \{{RSRawEncoder}} fallback.
> This is the Ozone-side equivalent of HADOOP-17209, which fixed the same class
> of leak in Hadoop HDFS's \{{ErasureCodingOutputStream}}.
> h3. Workaround
> Without a code fix, operators can:
> h1. Restart S3G periodically (e.g. daily via cron) to reset RSS.
> h1. Replace \{{NativeRSRawErasureCoderFactory}} /
> \{{NativeXORRawErasureCoderFactory}} in
> \{{META-INF/services/org.apache.ozone.erasurecode.rawcoder.RawErasureCoderFactory}}
> inside {{hdds-erasurecode-<version>.jar}} with the pure-Java
> \{{RSRawErasureCoderFactory}} / \{{XORRawErasureCoderFactory}}. This avoids
> the JNI allocation entirely but loses ISA-L acceleration.
> Neither workaround is ideal; a source-code fix is the proper resolution.
> h3. References
> * [HADOOP-17209|https://issues.apache.org/jira/browse/HADOOP-17209]: Erasure
> Coding: Native library memory leak (Hadoop-side fix that Ozone should mirror
> on \{{ECKeyOutputStream}})
> * Ozone source:
> \{{hadoop-ozone/client/src/main/java/org/apache/hadoop/ozone/client/io/ECKeyOutputStream.java}}
> * Ozone source:
> \{{hadoop-hdds/erasurecode/src/main/java/org/apache/ozone/erasurecode/rawcoder/NativeRSRawEncoder.java}}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
