[
https://issues.apache.org/jira/browse/HADOOP-20053?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Yuming Wang updated HADOOP-20053:
---------------------------------
Component/s: common
Description:
h2. Symptom
Spark executors with HDFS short-circuit reads enabled die with SIGSEGV. Two
production hs_err files show the *identical* crash site:
{code}
# Problematic frame:
# V [libjvm.so+0x8932b7] jni_NewObject+0x107
siginfo: si_addr: 0x0 (SEGV_MAPERR)
Registers: RSI=0x0, RDX=0x0 <- jclass and jmethodID arguments are NULL
{code}
Java stack of the crashing thread:
{code}
DomainSocket.receiveFileDescriptors0
DomainSocket.recvFileInputStreams
DfsClientShmManager$EndpointShmManager.requestNewShm
... ShortCircuitCache.allocShmSlot ... BlockReaderFactory.build
... ParquetFileReader.readFooter ... (Spark task reading HDFS)
{code}
Sibling executors that do not crash instead log:
{code}
java.lang.UnsupportedOperationException: NativeIO is not available.
at
org.apache.hadoop.hdfs.shortcircuit.ShortCircuitShm.<init>(ShortCircuitShm.java:470)
{code}
h2. Root cause chain (all links verified against production logs)
*1)* A version skew (newer Java code, older {{libhadoop.so}} on the nodes)
makes {{NativeIO.POSIX}} initialization fail. The failure is only logged at
DEBUG via PerformanceAdvisory, so production logs show nothing. With DEBUG
enabled we captured both directions of the skew:
{code}
DEBUG PerformanceAdvisory: Unable to initialize NativeIO libraries
java.lang.UnsatisfiedLinkError: 'void
...NativeIO$POSIX.initNativePosix(boolean)'
...
DEBUG PerformanceAdvisory: Unable to initialize NativeIO libraries
java.lang.NoSuchFieldError: workaroundNonThreadSafePasswdCalls
{code}
*2)* The native {{initNative*}} *error paths call {{fd_deinit(env)}}* ("these
are all idempotent and safe to call even if the class wasn't inited yet"). That
comment is wrong for the FileDescriptor cache: {{fd_class}}/{{fd_constructor}}
are *shared with DomainSocket*, which may have initialized them already through
{{DomainSocket.anchorNative()}} (a once-per-JVM call). After the teardown they
are NULL forever - {{fd_init()}} is never re-invoked.
*3)* Whether a given JVM crashes or only warns depends on *class initialization
order*: if {{DomainSocket.<clinit>}} ran before the NativeIO failure, the cache
is poisoned and the next {{receiveFileDescriptors0}} executes {{fd_create()}}
-> {{NewObject(env, NULL, NULL)}} -> SIGSEGV. If {{DomainSocket.<clinit>}} runs
after the failure, its {{fd_init()}} re-populates the cache and the same
request fails gracefully with "NativeIO is not available".
*4)* {{fd_create()}}/{{fd_get()}} had no NULL guards, so any JVM crashes at the
same PC deterministically. This is JVM-independent.
Note: we initially suspected JVM-side GC bugs (G1 pinning) and ruled them out -
the registers show clean NULLs, not heap corruption.
h2. Proposed fix
* {{fd_init()}}: resolve all members into locals first and publish the statics
atomically; failures leave the statics NULL and are retry-friendly.
* {{fd_get()}}/{{fd_create()}} (unix and windows): lazily re-initialize the
cache, then throw IllegalStateException instead of calling JNI with NULL
members. The existing {{receiveFileDescriptors0}} error path propagates it and
the client falls back to a regular block reader.
* {{NativeIO.c}}: remove {{fd_deinit(env)}} from the three init error paths
(the FileDescriptor cache must not be torn down by unrelated NativeIO
failures). {{fd_deinit()}} itself is retained for symmetry / explicit shutdown
use.
* {{NativeIO.java}}: log the init failure at WARN instead of DEBUG through a
shared constant message (kept a constant expression so javac folds it into the
inner classes whose static initializers can run re-entrantly).
Known tradeoff (documented in source and commit message): {{fd_init()}} is
intentionally not thread-safe; a cold-cache race can leak at most one global
ref per racing thread and is caught by the NULL checks, so the worst outcome is
a spurious IllegalStateException. fd_create() is only reached from
short-circuit shm setup which is near-serialized (DfsClientShmManager loading
lock).
h2. Testing
No unit test added: reproducing requires a version-skewed libhadoop.so, which
cannot be constructed in a unit test. Failure is evidenced by the production
hs_err files above; post-fix behavior (graceful IllegalStateException through
the existing DomainSocket error path) is verified by inspection; end-to-end
verification plan: deploy a rebuilt libhadoop.so to the skewed environment and
observe the WARN plus read fallback instead of the SIGSEGV. Existing
TestNativeIO / TestDomainSocket suites continue to pass.
h2. Mitigation for affected deployments
* {{dfs.client.read.shortcircuit=false}}, or
* align the node-side {{libhadoop.so}} with the Java code version (the root
trigger in this incident).
was:
h2. Symptom
Spark executors with HDFS short-circuit reads enabled die with SIGSEGV. Two
production hs_err files show the *identical* crash site:
{code}
# Problematic frame:
# V [libjvm.so+0x8932b7] jni_NewObject+0x107
siginfo: si_addr: 0x0 (SEGV_MAPERR)
Registers: RSI=0x0, RDX=0x0 <- jclass and jmethodID arguments are NULL
{code}
Java stack of the crashing thread:
{code}
DomainSocket.receiveFileDescriptors0
DomainSocket.recvFileInputStreams
DfsClientShmManager$EndpointShmManager.requestNewShm
... ShortCircuitCache.allocShmSlot ... BlockReaderFactory.build
... ParquetFileReader.readFooter ... (Spark task reading HDFS)
{code}
Sibling executors that do not crash instead log:
{code}
java.lang.UnsupportedOperationException: NativeIO is not available.
at
org.apache.hadoop.hdfs.shortcircuit.ShortCircuitShm.<init>(ShortCircuitShm.java:470)
{code}
h2. Root cause chain (all links verified against production logs)
*1)* A version skew (newer Java code, older {{libhadoop.so}} on the nodes)
makes {{NativeIO.POSIX}} initialization fail. The failure is only logged at
DEBUG via PerformanceAdvisory, so production logs show nothing. With DEBUG
enabled we captured both directions of the skew:
{code}
DEBUG PerformanceAdvisory: Unable to initialize NativeIO libraries
java.lang.UnsatisfiedLinkError: 'void
...NativeIO$POSIX.initNativePosix(boolean)'
...
DEBUG PerformanceAdvisory: Unable to initialize NativeIO libraries
java.lang.NoSuchFieldError: workaroundNonThreadSafePasswdCalls
{code}
*2)* The native {{initNative*}} *error paths call {{fd_deinit(env)}}* ("these
are all idempotent and safe to call even if the class wasn't inited yet"). That
comment is wrong for the FileDescriptor cache: {{fd_class}}/{{fd_constructor}}
are *shared with DomainSocket*, which may have initialized them already through
{{DomainSocket.anchorNative()}} (a once-per-JVM call). After the teardown they
are NULL forever - {{fd_init()}} is never re-invoked.
*3)* Whether a given JVM crashes or only warns depends on *class initialization
order*: if {{DomainSocket.<clinit>}} ran before the NativeIO failure, the cache
is poisoned and the next {{receiveFileDescriptors0}} executes {{fd_create()}}
-> {{NewObject(env, NULL, NULL)}} -> SIGSEGV. If {{DomainSocket.<clinit>}} runs
after the failure, its {{fd_init()}} re-populates the cache and the same
request fails gracefully with "NativeIO is not available".
*4)* {{fd_create()}}/{{fd_get()}} had no NULL guards, so any JVM crashes at the
same PC deterministically. This is JVM-independent.
Note: we initially suspected JVM-side GC bugs (G1 pinning) and ruled them out -
the registers show clean NULLs, not heap corruption.
h2. Proposed fix
* {{fd_init()}}: resolve all members into locals first and publish the statics
atomically; failures leave the statics NULL and are retry-friendly.
* {{fd_get()}}/{{fd_create()}} (unix and windows): lazily re-initialize the
cache, then throw IllegalStateException instead of calling JNI with NULL
members. The existing {{receiveFileDescriptors0}} error path propagates it and
the client falls back to a regular block reader.
* {{NativeIO.c}}: remove {{fd_deinit(env)}} from the three init error paths
(the FileDescriptor cache must not be torn down by unrelated NativeIO
failures). {{fd_deinit()}} itself is retained for symmetry / explicit shutdown
use.
* {{NativeIO.java}}: log the init failure at WARN instead of DEBUG through a
shared constant message (kept a constant expression so javac folds it into the
inner classes whose static initializers can run re-entrantly).
Known tradeoff (documented in source and commit message): {{fd_init()}} is
intentionally not thread-safe; a cold-cache race can leak at most one global
ref per racing thread and is caught by the NULL checks, so the worst outcome is
a spurious IllegalStateException. fd_create() is only reached from
short-circuit shm setup which is near-serialized (DfsClientShmManager loading
lock).
h2. Testing
No unit test added: reproducing requires a version-skewed libhadoop.so, which
cannot be constructed in a unit test. Failure is evidenced by the production
hs_err files above; post-fix behavior (graceful IllegalStateException through
the existing DomainSocket error path) is verified by inspection; end-to-end
verification plan: deploy a rebuilt libhadoop.so to the skewed environment and
observe the WARN plus read fallback instead of the SIGSEGV. Existing
TestNativeIO / TestDomainSocket suites continue to pass.
h2. Mitigation for affected deployments
* {{dfs.client.read.shortcircuit=false}}, or
* align the node-side {{libhadoop.so}} with the Java code version (the root
trigger in this incident).
> SIGSEGV in jni_NewObject after a failed NativeIO initialization poisons the
> shared FileDescriptor natives cache
> ---------------------------------------------------------------------------------------------------------------
>
> Key: HADOOP-20053
> URL: https://issues.apache.org/jira/browse/HADOOP-20053
> Project: Hadoop Common
> Issue Type: Bug
> Components: common
> Affects Versions: 3.5.0
> Reporter: Yuming Wang
> Priority: Major
> Labels: pull-request-available
>
> h2. Symptom
> Spark executors with HDFS short-circuit reads enabled die with SIGSEGV. Two
> production hs_err files show the *identical* crash site:
> {code}
> # Problematic frame:
> # V [libjvm.so+0x8932b7] jni_NewObject+0x107
> siginfo: si_addr: 0x0 (SEGV_MAPERR)
> Registers: RSI=0x0, RDX=0x0 <- jclass and jmethodID arguments are NULL
> {code}
> Java stack of the crashing thread:
> {code}
> DomainSocket.receiveFileDescriptors0
> DomainSocket.recvFileInputStreams
> DfsClientShmManager$EndpointShmManager.requestNewShm
> ... ShortCircuitCache.allocShmSlot ... BlockReaderFactory.build
> ... ParquetFileReader.readFooter ... (Spark task reading HDFS)
> {code}
> Sibling executors that do not crash instead log:
> {code}
> java.lang.UnsupportedOperationException: NativeIO is not available.
> at
> org.apache.hadoop.hdfs.shortcircuit.ShortCircuitShm.<init>(ShortCircuitShm.java:470)
> {code}
> h2. Root cause chain (all links verified against production logs)
> *1)* A version skew (newer Java code, older {{libhadoop.so}} on the nodes)
> makes {{NativeIO.POSIX}} initialization fail. The failure is only logged at
> DEBUG via PerformanceAdvisory, so production logs show nothing. With DEBUG
> enabled we captured both directions of the skew:
> {code}
> DEBUG PerformanceAdvisory: Unable to initialize NativeIO libraries
> java.lang.UnsatisfiedLinkError: 'void
> ...NativeIO$POSIX.initNativePosix(boolean)'
> ...
> DEBUG PerformanceAdvisory: Unable to initialize NativeIO libraries
> java.lang.NoSuchFieldError: workaroundNonThreadSafePasswdCalls
> {code}
> *2)* The native {{initNative*}} *error paths call {{fd_deinit(env)}}* ("these
> are all idempotent and safe to call even if the class wasn't inited yet").
> That comment is wrong for the FileDescriptor cache:
> {{fd_class}}/{{fd_constructor}} are *shared with DomainSocket*, which may
> have initialized them already through {{DomainSocket.anchorNative()}} (a
> once-per-JVM call). After the teardown they are NULL forever - {{fd_init()}}
> is never re-invoked.
> *3)* Whether a given JVM crashes or only warns depends on *class
> initialization order*: if {{DomainSocket.<clinit>}} ran before the NativeIO
> failure, the cache is poisoned and the next {{receiveFileDescriptors0}}
> executes {{fd_create()}} -> {{NewObject(env, NULL, NULL)}} -> SIGSEGV. If
> {{DomainSocket.<clinit>}} runs after the failure, its {{fd_init()}}
> re-populates the cache and the same request fails gracefully with "NativeIO
> is not available".
> *4)* {{fd_create()}}/{{fd_get()}} had no NULL guards, so any JVM crashes at
> the same PC deterministically. This is JVM-independent.
> Note: we initially suspected JVM-side GC bugs (G1 pinning) and ruled them out
> - the registers show clean NULLs, not heap corruption.
> h2. Proposed fix
> * {{fd_init()}}: resolve all members into locals first and publish the
> statics atomically; failures leave the statics NULL and are retry-friendly.
> * {{fd_get()}}/{{fd_create()}} (unix and windows): lazily re-initialize the
> cache, then throw IllegalStateException instead of calling JNI with NULL
> members. The existing {{receiveFileDescriptors0}} error path propagates it
> and the client falls back to a regular block reader.
> * {{NativeIO.c}}: remove {{fd_deinit(env)}} from the three init error paths
> (the FileDescriptor cache must not be torn down by unrelated NativeIO
> failures). {{fd_deinit()}} itself is retained for symmetry / explicit
> shutdown use.
> * {{NativeIO.java}}: log the init failure at WARN instead of DEBUG through a
> shared constant message (kept a constant expression so javac folds it into
> the inner classes whose static initializers can run re-entrantly).
> Known tradeoff (documented in source and commit message): {{fd_init()}} is
> intentionally not thread-safe; a cold-cache race can leak at most one global
> ref per racing thread and is caught by the NULL checks, so the worst outcome
> is a spurious IllegalStateException. fd_create() is only reached from
> short-circuit shm setup which is near-serialized (DfsClientShmManager loading
> lock).
> h2. Testing
> No unit test added: reproducing requires a version-skewed libhadoop.so, which
> cannot be constructed in a unit test. Failure is evidenced by the production
> hs_err files above; post-fix behavior (graceful IllegalStateException through
> the existing DomainSocket error path) is verified by inspection; end-to-end
> verification plan: deploy a rebuilt libhadoop.so to the skewed environment
> and observe the WARN plus read fallback instead of the SIGSEGV. Existing
> TestNativeIO / TestDomainSocket suites continue to pass.
> h2. Mitigation for affected deployments
> * {{dfs.client.read.shortcircuit=false}}, or
> * align the node-side {{libhadoop.so}} with the Java code version (the root
> trigger in this incident).
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
