Yuming Wang created HADOOP-20053:
------------------------------------
Summary: SIGSEGV in jni_NewObject after a failed NativeIO
initialization poisons the shared FileDescriptor natives cache
Key: HADOOP-20053
URL: https://issues.apache.org/jira/browse/HADOOP-20053
Project: Hadoop Common
Issue Type: Bug
Affects Versions: 3.5.0
Reporter: Yuming Wang
h2. Symptom
Spark executors with HDFS short-circuit reads enabled die with SIGSEGV. Two
production hs_err files show the *identical* crash site:
{code}
# Problematic frame:
# V [libjvm.so+0x8932b7] jni_NewObject+0x107
siginfo: si_addr: 0x0 (SEGV_MAPERR)
Registers: RSI=0x0, RDX=0x0 <- jclass and jmethodID arguments are NULL
{code}
Java stack of the crashing thread:
{code}
DomainSocket.receiveFileDescriptors0
DomainSocket.recvFileInputStreams
DfsClientShmManager$EndpointShmManager.requestNewShm
... ShortCircuitCache.allocShmSlot ... BlockReaderFactory.build
... ParquetFileReader.readFooter ... (Spark task reading HDFS)
{code}
Sibling executors that do not crash instead log:
{code}
java.lang.UnsupportedOperationException: NativeIO is not available.
at
org.apache.hadoop.hdfs.shortcircuit.ShortCircuitShm.<init>(ShortCircuitShm.java:470)
{code}
h2. Root cause chain (all links verified against production logs)
*1)* A version skew (newer Java code, older {{libhadoop.so}} on the nodes)
makes {{NativeIO.POSIX}} initialization fail. The failure is only logged at
DEBUG via PerformanceAdvisory, so production logs show nothing. With DEBUG
enabled we captured both directions of the skew:
{code}
DEBUG PerformanceAdvisory: Unable to initialize NativeIO libraries
java.lang.UnsatisfiedLinkError: 'void
...NativeIO$POSIX.initNativePosix(boolean)'
...
DEBUG PerformanceAdvisory: Unable to initialize NativeIO libraries
java.lang.NoSuchFieldError: workaroundNonThreadSafePasswdCalls
{code}
*2)* The native {{initNative*}} *error paths call {{fd_deinit(env)}}* ("these
are all idempotent and safe to call even if the class wasn't inited yet"). That
comment is wrong for the FileDescriptor cache: {{fd_class}}/{{fd_constructor}}
are *shared with DomainSocket*, which may have initialized them already through
{{DomainSocket.anchorNative()}} (a once-per-JVM call). After the teardown they
are NULL forever - {{fd_init()}} is never re-invoked.
*3)* Whether a given JVM crashes or only warns depends on *class initialization
order*: if {{DomainSocket.<clinit>}} ran before the NativeIO failure, the cache
is poisoned and the next {{receiveFileDescriptors0}} executes {{fd_create()}}
-> {{NewObject(env, NULL, NULL)}} -> SIGSEGV. If {{DomainSocket.<clinit>}} runs
after the failure, its {{fd_init()}} re-populates the cache and the same
request fails gracefully with "NativeIO is not available".
*4)* {{fd_create()}}/{{fd_get()}} had no NULL guards, so any JVM crashes at the
same PC deterministically. This is JVM-independent.
Note: we initially suspected JVM-side GC bugs (G1 pinning) and ruled them out -
the registers show clean NULLs, not heap corruption.
h2. Proposed fix
* {{fd_init()}}: resolve all members into locals first and publish the statics
atomically; failures leave the statics NULL and are retry-friendly.
* {{fd_get()}}/{{fd_create()}} (unix and windows): lazily re-initialize the
cache, then throw IllegalStateException instead of calling JNI with NULL
members. The existing {{receiveFileDescriptors0}} error path propagates it and
the client falls back to a regular block reader.
* {{NativeIO.c}}: remove {{fd_deinit(env)}} from the three init error paths
(the FileDescriptor cache must not be torn down by unrelated NativeIO
failures). {{fd_deinit()}} itself is retained for symmetry / explicit shutdown
use.
* {{NativeIO.java}}: log the init failure at WARN instead of DEBUG through a
shared constant message (kept a constant expression so javac folds it into the
inner classes whose static initializers can run re-entrantly).
Known tradeoff (documented in source and commit message): {{fd_init()}} is
intentionally not thread-safe; a cold-cache race can leak at most one global
ref per racing thread and is caught by the NULL checks, so the worst outcome is
a spurious IllegalStateException. fd_create() is only reached from
short-circuit shm setup which is near-serialized (DfsClientShmManager loading
lock).
h2. Testing
No unit test added: reproducing requires a version-skewed libhadoop.so, which
cannot be constructed in a unit test. Failure is evidenced by the production
hs_err files above; post-fix behavior (graceful IllegalStateException through
the existing DomainSocket error path) is verified by inspection; end-to-end
verification plan: deploy a rebuilt libhadoop.so to the skewed environment and
observe the WARN plus read fallback instead of the SIGSEGV. Existing
TestNativeIO / TestDomainSocket suites continue to pass.
h2. Mitigation for affected deployments
* {{dfs.client.read.shortcircuit=false}}, or
* align the node-side {{libhadoop.so}} with the Java code version (the root
trigger in this incident).
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
