Meng-Shuan Tsai created HDDS-16235:
--------------------------------------
Summary: Intermittent failure in
TestOzoneClientRetriesOnExceptions#testMaxRetriesByOzoneClient
Key: HDDS-16235
URL: https://issues.apache.org/jira/browse/HDDS-16235
Project: Apache Ozone
Issue Type: Sub-task
Reporter: Meng-Shuan Tsai
Assignee: Meng-Shuan Tsai
testMaxRetriesByOzoneClient fails intermittently (~3% of runs) at
TestOzoneClientRetriesOnExceptions.java:206 with:
{code:java}
java.io.IOException: Failed to execute command cmdType: CreateContainer
containerID: 1 datanodeUuid: "..." createContainer { state: OPEN }
at
org.apache.hadoop.hdds.scm.XceiverClientSpi.getIOExceptionForSendCommand(XceiverClientSpi.java:178)
at
org.apache.hadoop.hdds.scm.XceiverClientSpi.sendCommand(XceiverClientSpi.java:131)
at
org.apache.hadoop.ozone.client.rpc.TestOzoneClientRetriesOnExceptions.testMaxRetriesByOzoneClient(TestOzoneClientRetriesOnExceptions.java:206)
Caused by: java.util.concurrent.ExecutionException:
org.apache.ratis.protocol.exceptions.StateMachineException:
org.apache.hadoop.hdds.scm.container.common.helpers.StorageContainerException:
Container creation failed because key value container already exists
at
org.apache.hadoop.hdds.scm.XceiverClientSpi.sendCommand(XceiverClientSpi.java:125)
Caused by: org.apache.ratis.protocol.exceptions.StateMachineException:
org.apache.hadoop.hdds.scm.container.common.helpers.StorageContainerException:
Container creation failed because key value container already exists
at
org.apache.hadoop.ozone.container.common.transport.server.ratis.ContainerStateMachine.lambda$applyTransaction$2(ContainerStateMachine.java:1193)
{code}
Steps to reproduce: run TestOzoneClientRetriesOnExceptions out of 100 runs (
[https://github.com/shuan1026/ozone/actions/runs/32260061359]), 3 failed with
this identical stack trace, all on containerID 1. Datanode-side log for one
such failure:
{code:java}
14:18:42,671 leader elected for the container's pipeline (term 1)
14:18:42,788 CreateContainer for containerID 1 fails: Result:
CONTAINER_ALREADY_EXISTS
14:18:42,800 ContainerStateMachine ApplyTransaction failed at logIndex 5
14:18:42,807 pipeline close action triggered as a result
{code}
Line 206 is the test's own manual
xceiverClient.sendCommand(getCreateContainerRequest(...)), sent once per
pre-allocated container to ensure it is OPEN before writing. By design this
should be the only path that creates the container on the datanode, but the log
shows it was already created by something else ~117ms after the pipeline's
leader election, so the test's own call is rejected as a duplicate,
non-idempotent create.
No @Flaky annotation exists on this test and no prior JIRA report was found for
it.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
