Wei-Chiu Chuang created HDDS-17254:
--------------------------------------
Summary: Client-side non-daemon executors and timers can prevent
JVM shutdown
Key: HDDS-17254
URL: https://issues.apache.org/jira/browse/HDDS-17254
Project: Apache Ozone
Issue Type: Bug
Components: Ozone Client, EC Client
Reporter: Wei-Chiu Chuang
h2. Problem
HDDS-10564 / [PR #6418|https://github.com/apache/ozone/pull/6418] made
RpcClient output-stream executors use daemon threads so an unclosed
FileSystem/client would not keep a command-line JVM alive. A source audit found
four other client-side executor/timer creation sites with the same potential
shutdown risk.
This report is based on source inspection at
d0949f291791c399f3e54409fcb92656a80597a4. JVM-exit hangs have not yet been
reproduced in a subprocess test. Not every invocation starts the relevant
worker: the conditions for each path are listed below.
h2. Affected paths
h3. 1. Ratis streaming writes: BlockDataStreamOutput
The constructor creates its response executor with
Executors.newSingleThreadExecutor(), whose default thread factory creates
non-daemon threads. This is separate from RpcClient.writeExecutor, so
HDDS-10564 does not cover it. After an asynchronous response starts the worker,
an abandoned/unclosed streaming output can keep the JVM alive. Normal close
calls cleanup in finally, and cleanup shuts down the executor.
* Executor creation:
https://github.com/apache/ozone/blob/d0949f291791c399f3e54409fcb92656a80597a4/hadoop-hdds/client/src/main/java/org/apache/hadoop/hdds/scm/storage/BlockDataStreamOutput.java#L195
* Normal close/cleanup:
https://github.com/apache/ozone/blob/d0949f291791c399f3e54409fcb92656a80597a4/hadoop-hdds/client/src/main/java/org/apache/hadoop/hdds/scm/storage/BlockDataStreamOutput.java#L689
* Executor shutdown:
https://github.com/apache/ozone/blob/d0949f291791c399f3e54409fcb92656a80597a4/hadoop-hdds/client/src/main/java/org/apache/hadoop/hdds/scm/storage/BlockDataStreamOutput.java#L736
h3. 2. Short-circuit reads: per-connection timer
XceiverClientShortCircuit.connect() creates new Timer(prefix + "-Timer")
without the daemon argument. The timer thread starts when the connection is
established. If client cleanup is omitted, it can keep the JVM alive. Normal
close cancels the timer.
*
https://github.com/apache/ozone/blob/d0949f291791c399f3e54409fcb92656a80597a4/hadoop-hdds/client/src/main/java/org/apache/hadoop/hdds/scm/XceiverClientShortCircuit.java#L143
h3. 3. Short-circuit reads: singleton DomainSocketFactory timer
When short-circuit reads are configured and native support is available,
DomainSocketFactory creates new Timer(name) without the daemon argument. This
is a second non-daemon timer, separate from the connection timer above. Factory
close cancels it, but an unclosed client/factory can leave it alive.
* Creation:
https://github.com/apache/ozone/blob/d0949f291791c399f3e54409fcb92656a80597a4/hadoop-hdds/client/src/main/java/org/apache/hadoop/hdds/scm/storage/DomainSocketFactory.java#L160
* Cancellation:
https://github.com/apache/ozone/blob/d0949f291791c399f3e54409fcb92656a80597a4/hadoop-hdds/client/src/main/java/org/apache/hadoop/hdds/scm/storage/DomainSocketFactory.java#L272
h3. 4. Default low-level EC reader factory
BlockInputStreamFactoryImpl's no-argument constructor supplies
Executors::newSingleThreadExecutor. If an EC reconstruction reader is
instantiated through this default factory and executes reconstruction work, it
can start non-daemon workers. The reconstruction stream's close method does not
shut down the supplied executor. The normal RpcClient path supplies its own
daemon executors and is not affected by this default-factory creation site.
* Default factory:
https://github.com/apache/ozone/blob/d0949f291791c399f3e54409fcb92656a80597a4/hadoop-hdds/client/src/main/java/org/apache/hadoop/ozone/client/io/BlockInputStreamFactoryImpl.java#L70
* Reconstruction executor acquisition:
https://github.com/apache/ozone/blob/d0949f291791c399f3e54409fcb92656a80597a4/hadoop-hdds/client/src/main/java/org/apache/hadoop/ozone/client/io/ECBlockInputStreamFactoryImpl.java#L95
* Reconstruction stream close:
https://github.com/apache/ozone/blob/d0949f291791c399f3e54409fcb92656a80597a4/hadoop-hdds/client/src/main/java/org/apache/hadoop/ozone/client/io/ECBlockReconstructedStripeInputStream.java#L738
* A production caller of the default factory is the checksum-verification tool:
https://github.com/apache/ozone/blob/d0949f291791c399f3e54409fcb92656a80597a4/hadoop-ozone/cli-debug/src/main/java/org/apache/hadoop/ozone/debug/replicas/ChecksumVerifier.java#L61
h2. Expected behavior / suggested fix
Client housekeeping workers should not prevent JVM termination solely because a
caller has finished its work but omitted explicit cleanup. Apply the
daemon-thread principle used by HDDS-10564 to the affected background
executors/timers, while preserving normal close/cancel behavior.
* Use a named daemon ThreadFactory for the streaming response executor.
* Use daemon timers for both short-circuit timer sites.
* Give the default EC reconstruction executor an explicit lifecycle/owner and
daemon threads. Do not indiscriminately shut down caller-supplied shared
executors from an individual stream's close method.
h2. Validation needed
Add isolated subprocess tests with bounded exit timeouts:
# Start streaming response work, omit stream/client close, return from main,
and verify the JVM exits.
# Establish a short-circuit connection, omit client/factory close, return from
main, and verify both timer threads do not retain the JVM (requires native
short-circuit prerequisites).
# Trigger EC reconstruction through the no-argument factory, close the stream,
and verify no orphan executor keeps the JVM alive; also check the
abandoned-stream case.
# Verify explicit cleanup still terminates/releases the owned resources and
does not break shared executors.
Daemon workers do not replace application responsibility to flush and close
streams. No data-durability guarantee is inferred from changing thread daemon
status.
Related prior fix: HDDS-10564.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
