smengcl opened a new pull request, #11454: URL: https://github.com/apache/ozone/pull/11454
Generated-by: Claude Code (Opus 5.5) ## What changes were proposed in this pull request? `XceiverClientManager` closes an evicted client on whichever thread triggers the eviction, while that thread holds the `clientCache` lock. For `XceiverClientRatis` the close waits for gRPC channel termination inside `RaftClient.close()`, up to 3 s gracefully plus 2 s forced per channel. A writer starting a new block was seen blocked for about 3 s this way, stalling every in flight HBase WAL sync, and any other `acquireClient` or `releaseClient` caller on the same manager would wait too. HDDS-15849 fixed this for `XceiverClientGrpc` only. Details and stack traces are in the Jira. This PR moves the close to a background thread owned by `XceiverClientManager`: * `XceiverClientSpi.setEvicted()` and `decrementReference()` no longer call `close()`. They return whether the client should now be closed, and the manager hands it to a single daemon thread. This covers the removal listener and both `releaseClient` paths. * Closes still run one at a time, as they effectively did under the lock. The thread exits after 60 s idle. * Pending closes are bounded by `scm.container.client.max.size`. Beyond that the client is closed by the caller as before, which keeps backpressure. * `XceiverClientManager.close()` waits for pending closes. * A failure in `close()` of an evicted client is now logged at WARN instead of propagating to the thread that released it. No config, API or wire change. ## What is the link to the Apache JIRA https://issues.apache.org/jira/browse/HDDS-16812 ## How was this patch tested? * New `TestXceiverClientManagerCloseContention#slowCloseOfEvictedClientDoesNotBlockCaller`: evicts a client whose `close()` blocks and asserts the close runs off the caller thread, without the cache lock, and that closing the manager waits for it. It fails without the fix. * Existing `evictionDrivenCloseDoesNotSerializeAcquisition` now includes draining the pending closes in the measured time, so it still catches a slow `XceiverClientGrpc.close()` (checked by adding a 100 ms sleep there). * `TestXceiverClientManager#testFreeByReference` and `testFreeByEviction` now wait for the asynchronous close before asserting that the client is closed. * `hdds-client` unit tests, the `XceiverClientManager` tests of `TestOzoneIntegrationNonHA`, and checkstyle pass locally. * The HBase WAL workload that showed the stall was not rerun with this patch. 🤖 Generated with [Claude Code](https://claude.com/claude-code) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
