Andrey Yarovoy created HDDS-16596:
-------------------------------------
Summary: Isolate synchronous Ratis-commit and SCM RPC waits from
the OM client handler pool so a transient bucket-lock convoy cannot escalate to
full-pool exhaustion
Key: HDDS-16596
URL: https://issues.apache.org/jira/browse/HDDS-16596
Project: Apache Ozone
Issue Type: Bug
Components: OM
Reporter: Andrey Yarovoy
h2. Problem
The OM services client RPCs on a single shared IPC handler pool
({{{}ozone.om.handler.count.key{}}}). Write requests block their handler thread
synchronously for the duration of the Ratis commit —
{{OzoneManagerRatisServer.submitRequestToRatis}} parks the handler in
{{CompletableFuture.get()}} until the write is replicated — and outbound SCM
calls ({{{}allocateBlock{}}}, container/pipeline location) likewise park their
handler for the whole round trip inside {{RetryInvocationHandler}} /
{{{}Client.getRpcResponse{}}}.
Because these waits sit on the same pool that serves reads, the pool has no
reserved headroom. Under sustained write load a large fraction of handlers are
permanently parked on commit, so the pool runs at near-zero idle at rest. When
a transient contention event then arrives — for example a bucket
read/write-lock convoy that would otherwise drain in microseconds — the
arriving requests consume the few remaining handlers and the localized stall
escalates into a full-pool outage affecting unrelated buckets and read RPCs.
h2. Evidence
On the heavy loaded cluster with mixed IO patterns per-thread analysis of OM
stack dumps from a saturated OM (200-thread handler pool) versus a healthy OM
under comparable client load:
The two profiles carry the same peak lock-convoy magnitude (145 vs 133
threads), so the convoy itself is not the differentiator. The differentiator is
headroom: the saturated OM held ~93% of its handlers in commit waits with zero
idle, so a convoy had nothing to absorb it; the healthy OM held ~93% of
handlers idle and absorbed convoys of similar size without a pool-wide stall.
h2. Proposed direction
Decouple the blocking waits from the client-facing handler pool so a transient
convoy cannot consume the whole pool:
* Complete the Ratis commit asynchronously (release the handler once the
request is submitted, respond on commit) rather than blocking the handler in
{{{}get(){}}}, *or* run commit waits on a bounded executor separate from the
handler pool.
* Isolate outbound SCM RPCs onto a bounded pool with backpressure so a slow
SCM cannot exhaust client handlers.
* As a floor, ensure read RPCs retain reserved capacity independent of
in-flight write commits ({{{}ozone.om.read.threadpool{}}}, default 10, is small
relative to the handler pool).
Each has correctness/backpressure implications (in-flight accounting,
retry/failover, ordering) and should be scoped as design subtasks.
h2. Testing
* A concurrency benchmark that saturates the write path and confirms read-RPC
latency and idle-handler count stay bounded while commits are in flight.
* A fault-injection test with a deliberately slow downstream (Ratis or SCM)
asserting the handler pool retains headroom rather than reaching full occupancy.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
