yandrey321 opened a new pull request, #10747:
URL: https://github.com/apache/ozone/pull/10747
## What changes were proposed in this pull request?
XceiverClientGrpc.close() performs a blocking graceful channel shutdown, and
because clients are torn down through XceiverClientManager's cache eviction,
that blocking wait runs while the clientCache monitor is held — serializing
every concurrent acquireClient() / releaseClient() call.
HDDS-14571 changed close() from a forced shutdownNow() (which cancels
in-flight RPCs and terminates in ~1 ms) to a graceful shutdown() followed by a
polling loop that always sleeps at least SHUTDOWN_WAIT_INTERVAL_MILLIS (100 ms)
before it can observe termination:
```
for (ChannelInfo channelInfo : dnChannelInfoMap.values()) {
channelInfo.getChannel().shutdown();
}
while (!nonTerminatedChannels.isEmpty() && System.nanoTime() < deadline) {
nonTerminatedChannels.removeIf(ManagedChannel::isTerminated);
Thread.sleep(SHUTDOWN_WAIT_INTERVAL_MILLIS); // >= 100ms floor per close()
}
```
Every close() now costs ≥100 ms instead of ~1 ms. The eviction path that
invokes it holds the cache monitor throughout:
```
acquireClient() // synchronized (clientCache)
-> getClient()
-> clientCache.get(key, newClient)
-> Guava evicts an entry on the calling thread
-> RemovalListener.onRemoval() // synchronized (clientCache)
-> XceiverClientSpi.setEvicted() -> cleanup() -> close() //
blocks here
```
So concurrent readers that create/evict short-lived clients (e.g. EC
read/checksum collection, which uses a distinct client per placement group)
serialize behind a ≥100 ms lock hold per closed channel and degrade
non-linearly with concurrency.
Fix: restore immediate termination in close() — use channel.shutdownNow()
followed by channel.awaitTermination(SHUTDOWN_WAIT_MAX_SECONDS, ...), and drop
the mandatory-sleep polling loop. This keeps HDDS-14571's goal (no synchronized
methods on XceiverClientGrpc) while eliminating the blocking wait under the
cache lock.
Co authored with Claude Code
## What is the link to the Apache JIRA
https://issues.apache.org/jira/browse/HDDS-15849
## How was this patch tested?
Added TestXceiverClientManagerCloseContention, a unit test that exercises
the real XceiverClientManager and XceiverClientGrpc.close() (real
lazily-connected plaintext gRPC channels to random local datanode addresses; no
MiniOzoneCluster and no live datanode required, since close() only shuts
channels down). maxCacheSize=1 plus a distinct pipeline per iteration forces an
eviction-driven close on nearly every acquisition; 8 threads × 8 cycles produce
63 closes. It measures average wall-clock per close and asserts it stays well
under the pre-fix ~100 ms floor.
The same test, unmodified, against the code before and after this change:
run | wall-clock | ms / close | result
-- | -- | -- | --
Before (graceful shutdown() + 100 ms poll) | 6874 ms | 109 ms | ❌ fails
After (shutdownNow() + awaitTermination) | 201 ms | 3 ms | ✅ passes
~34× faster per close
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
