Andrey Yarovoy created HDDS-17648:
-------------------------------------
Summary: Key write hangs when its Ratis client is closed while a
watch-for-commit is still pending
Key: HDDS-17648
URL: https://issues.apache.org/jira/browse/HDDS-17648
Project: Apache Ozone
Issue Type: Improvement
Components: Ozone Client
Reporter: Andrey Yarovoy
Since HDDS-17491, `TestKeyInputStream#readAfterReplication(boolean)[2]`
intermittently fails on master with
`readAfterReplication(boolean) timed out after 5 minutes` (integration
(client), run 41300247417 on 20da41e535).
Locally it fails in 2 of 6 runs on 20da41e535 and in 0 of 6 runs on the commit
before it (4ade0f1e9b). The test
thread is stuck writing the test key, not reading it.
**What happens** (from the test log of a failing local run):
1. The write goes to a pipeline that still includes the datanode the previous
test case stopped, so a
watch-for-commit fails (`ALL_COMMITTED way commit failed`,
`RaftRetryFailureException`), and
`BlockOutputStream.handleFlush(true)` fails ("Failed to flush").
2. `handleFlush(true)` calls `cleanup(false)` in its `finally` block, which
releases the Ratis client, while the watch
for the other putBlock of the same key is still being retried by that client.
3. The released client has no references left, and its `XceiverClientManager`
cache entry has already expired:
`expireAfterAccess` counts from the last cache access, which was the start of
the write. The periodic
`cleanUp()` added by HDDS-17491 removes the entry, and the client is closed.
4. The pending watch then fails:
```
GrpcClientRpc - client-3F9447C48097: Failed
RaftClientRequest:client-3F9447C48097->...
org.apache.ratis.protocol.exceptions.AlreadyClosedException:
client-3F9447C48097 is already CLOSED
XceiverClientRatis - ALL_COMMITTED way commit failed on pipeline ...
AlreadyClosedException: org.apache.ratis.client.impl.RaftClientImpl@144b857d is
closed
```
5. `XceiverClientRatis.watchForCommit` handles the failure by sending a
MAJORITY_COMMITTED watch on the same,
now closed, client, from inside its `exceptionally` handler. The future it
returns is never completed.
6. `BlockOutputStream.waitForAllPendingFlushes()` waits for all flush futures
with an untimed
`allPendingFlushFutures.get()`, so the write never returns. In the test, JUnit
interrupts it after 5 minutes;
the write then retries on a new block and succeeds.
Which of the two ways leaves the future incomplete is not confirmed yet: either
Ratis returns a watch future that
never completes on a closed client, or the call throws inside the
`exceptionally` handler, where the exception is
swallowed and `replyFuture` is never completed.
**Why it starts with HDDS-17491:** releasing a client while its async work is
still pending (step 2) predates
HDDS-17491. Before it, an expired entry was only removed during a later cache
access, which did not happen in this
window in a single-client test, so the client stayed open. HDDS-17491 removes
expired entries every idle threshold
(10 s by default), so the window is hit regularly. In a busy client, any cache
access after the threshold could
already trigger the same hang before HDDS-17491.
**Impact:** a key write that hits a failed datanode can hang indefinitely
instead of retrying on another block.
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
