yandrey321 opened a new pull request, #11785:
URL: https://github.com/apache/ozone/pull/11785
## What changes were proposed in this pull request?
**Problem.** `XceiverClientManager` caches datanode clients with
`expireAfterAccess(scm.container.client.idle.threshold)`
(default 10 s), and an evicted client is closed once no one holds it. Guava
never evicts on a timer, though. It
removes expired entries only during cache operations, and then only in the
segment the operation touches (4 by
default).
- **An idle JVM never closes its clients.** Their connections and threads
stay open until the manager is closed.
- **A busy JVM closes idle clients only by chance.** In a test with 16 idle
clients, one cache write after the
threshold closed 5 or 6 of them.
With Ratis data streaming this keeps the JVM from exiting (HDDS-17278). A
streaming client holds the shared Netty
worker group, whose threads are not daemon threads (RATIS-2781). Most Hadoop
applications don't close their
FileSystem; they rely on a shutdown hook, and that hook can't run while
those threads are alive. Every MapReduce task
that writes with data streaming therefore hangs until the AppMaster kills it
after 60 s.
**Implementation idea.** A daemon single-thread scheduler in
`XceiverClientManager` (`XceiverClientManager-cleanup-0`)
runs `clientCache.cleanUp()` once per idle threshold. Expired clients that
no one holds are then closed within one to
two thresholds of their last use, whether the cache is used or not. The
scheduler is shut down in `close()`.
**Key decisions.**
- **A daemon thread,** so the cleanup can't keep a JVM alive itself.
- **Our own thread, not Guava or a shared pool.** Guava's `CacheBuilder` has
no scheduler; its documentation
recommends calling `cleanUp()` from a scheduled executor. A client close
can block (gRPC shutdown, Ratis stream
flush), so it shouldn't run on a shared pool such as
`ForkJoinPool.commonPool()`. Caffeine's scheduler would mean a
new dependency.
- **The existing rules are unchanged:** a client that is still referenced
stays open until it is released, and the
idle threshold still decides when a client counts as idle.
- **Failures don't stop the task:** an exception from a close is logged, and
the next run continues.
**Side effects.**
- One extra daemon thread per `XceiverClientManager`, normally one per Ozone
client. It is idle except for a
`cleanUp()` call every threshold.
- Idle clients are now closed on the cleanup thread, so connections are
released that previously stayed open until
the client was closed. A client used again after being closed is
recreated, as already happens after an eviction.
- **HDDS-17278:** a JVM that wrote with data streaming and doesn't close its
FileSystem now exits on its own. With the
default settings that is about 12–22 s after its last write: the idle
threshold, up to one more threshold until the
next cleanup run, and Netty's 2 s quiet period. RATIS-2781 is needed for
an immediate exit.
Generated-by: Claude Code (Claude Opus 5.5)
## What is the link to the Apache JIRA
https://issues.apache.org/jira/browse/HDDS-17491
## How was this patch tested?
unit + integration tests.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
