Siyao Meng created HDDS-16801:
---------------------------------
Summary: Intermittent failure in
TestPerVolumePushReplication#testHealthyVolumeReplicationAfterVolumeFailure
Key: HDDS-16801
URL: https://issues.apache.org/jira/browse/HDDS-16801
Project: Apache Ozone
Issue Type: Sub-task
Components: Ozone Datanode, test
Reporter: Siyao Meng
Assignee: Siyao Meng
{code}
[ERROR] Tests run: 3, Failures: 0, Errors: 1, Skipped: 0, Time elapsed: 101.2 s
<<< FAILURE! -- in
org.apache.hadoop.ozone.container.replication.TestPerVolumePushReplication
[ERROR]
org.apache.hadoop.ozone.container.replication.TestPerVolumePushReplication.testHealthyVolumeReplicationAfterVolumeFailure
-- Time elapsed: 35.31 s <<< ERROR!
java.util.concurrent.TimeoutException:
Timed out waiting for condition. Thread diagnostics:
...
at
org.apache.ozone.test.GenericTestUtils.waitFor(GenericTestUtils.java:137)
at
org.apache.hadoop.ozone.container.replication.TestPerVolumePushReplication.queuePushAndWaitForContainer(TestPerVolumePushReplication.java:386)
at
org.apache.hadoop.ozone.container.replication.TestPerVolumePushReplication.testHealthyVolumeReplicationAfterVolumeFailure(TestPerVolumePushReplication.java:219)
{code}
The source datanode dropped the injected push command:
{code}
WARN statemachine.StateContext (StateContext.java:getNextCommand(767)) -
Detect and drop a SCMCommand replicateContainerCommand: cmdID: 1791499115402,
encodedToken: "", term: 0, ..., containerId=1000003, ... from stale leader SCM,
stale term 0, latest term 2.
{code}
h3. Root cause
testPushAndScmReplicationWithPerVolumeEnabled restarts a datanode at the end,
and testHealthyVolumeReplicationAfterVolumeFailure can pick that same datanode
as the push source. queueReplicationCommand stamps the command with the
datanode's own term via
{{context.getTermOfLeaderSCM().ifPresent(cmd::setTerm)}}. On a just restarted
datanode that term is not initialized yet, so the command keeps term 0. When
StateContext later initializes the term from the max term in its command queue
(2, from real SCM commands such as close pipeline), the term 0 push is dropped
as coming from a stale leader SCM, and the wait for the container on the target
times out.
h3. Fix
Stamp the command with the SCM leader term
({{getScmContext().getTermOfLeader()}}), as TestCloseContainerByPipeline and
TestDeleteContainerHandler do, so it never depends on whether the datanode has
learned the term yet.
- https://github.com/apache/ozone/actions/runs/37850535429/job/113566423731
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
