Andrey Yarovoy created HDDS-16398:
-------------------------------------
Summary: TestCommitWatcher#testReleaseBuffersOnException
intermittently fails with AlreadyClosedException (NotLeaderException on
freshly-allocated pipeline)
Key: HDDS-16398
URL: https://issues.apache.org/jira/browse/HDDS-16398
Project: Apache Ozone
Issue Type: Bug
Components: SCM Client
Reporter: Andrey Yarovoy
{{org.apache.hadoop.hdds.scm.storage.TestCommitWatcher#testReleaseBuffersOnException}}
fails intermittently in CI. The failure occurs during test setup — while
waiting on the _first_ {{putBlock}} response ({{{}future1.get(){}}} at
{{{}TestCommitWatcher.java:262{}}}), before the test reaches the datanode
shutdowns that exercise the behavior actually under test.
*Stack trace*
{code} java.util.concurrent.ExecutionException:
org.apache.ratis.protocol.exceptions.AlreadyClosedException:
SlidingWindow$Client:client-B2BDCC4ECA19->RAFT is closed. at
java.base/java.util.concurrent.CompletableFuture.get(CompletableFuture.java:2094)
at
org.apache.hadoop.hdds.scm.storage.TestCommitWatcher.testReleaseBuffersOnException(TestCommitWatcher.java:262)
Caused by: org.apache.ratis.protocol.exceptions.AlreadyClosedException:
SlidingWindow$Client:client-... is closed. at
org.apache.ratis.util.SlidingWindow$Client.alreadyClosed(SlidingWindow.java:409)
... Caused by: org.apache.ratis.protocol.exceptions.RaftRetryFailureException:
Failed RaftClientRequest: client-...->d62e25aa-...@group-..., cid=29, seq=1*,
Watch(0), null for 3 attempts with RequestTypeDependentRetryPolicy\{ ... } at
org.apache.ratis.client.impl.RaftClientImpl.noMoreRetries(RaftClientImpl.java:352)
Caused by: org.apache.ratis.protocol.exceptions.NotLeaderException: Server
d62e25aa-... is not the leader, suggested leader is:
59f5747a-...|127.0.0.1:15071 at
org.apache.ratis.client.impl.ClientProtoUtils.toRaftClientReply(ClientProtoUtils.java:402)
\{code}
*Root cause*
The test allocates a fresh RATIS THREE pipeline and immediately issues async
write/putBlock commands:
{code} ContainerWithPipeline container = storageContainerLocationClient
.allocateContainer(RATIS, THREE, OZONE); Pipeline pipeline =
container.getPipeline(); ... ratisClient.sendCommandAsync(writeChunkRequest);
XceiverClientReply reply = ratisClient.sendCommandAsync(putBlockRequest);
\{code}
{{cluster.waitForClusterToBeReady()}} (called in {{{}@BeforeEach{}}}) waits for
datanode registration and SCM readiness, but does *not* guarantee that the
newly-allocated Ratis group has completed leader election. When the write lands
on a node that is not yet the leader, Ratis returns {{{}NotLeaderException{}}}.
Under the aggressive per-test retry policy (short watch timeout, limited
attempts), the request exhausts its retries and the client is closed, surfacing
as {{AlreadyClosedException}} from {{{}future1.get(){}}}.
This is a timing/leader-election race in test setup, not a product defect in
the commit-watch path.
*Reproducibility*
Intermittent. Timing-dependent; does not reproduce reliably. This test has a
long history of the same class of flakiness — see related issues below
(HDDS-9766 reports the identical {{AlreadyClosedException}} signature).
*Suggested fix*
Wait for the allocated pipeline to have an elected leader before issuing writes
— e.g., block on pipeline readiness / leader election
({{{}XceiverClientRatis.watchForCommit{}}} on the leader, or a
{{{}waitForPipelineReady{}}}-style helper) after {{allocateContainer}} and
before the first {{{}sendCommandAsync{}}}, so setup no longer races election.
Apply the same guard to {{{}testReleaseBuffers{}}}, which shares the pattern.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
