Sergey Soldatov created HDDS-17638:
--------------------------------------
Summary: Intermittent failure in
TestReconAsPassiveScm.testDatanodeRegistrationAndReports (container check)
Key: HDDS-17638
URL: https://issues.apache.org/jira/browse/HDDS-17638
Project: Apache Ozone
Issue Type: Bug
Components: Ozone Recon, test
Affects Versions: 2.2.2
Reporter: Sergey Soldatov
Assignee: Sergey Soldatov
{code:java}
org.opentest4j.AssertionFailedError: expected: <[ContainerInfo{id=#1,
state=OPEN, stateEnterTime=2026-10-31T05:23:15.250Z,
pipelineID=Pipeline-98a485ed-6453-4023-a3b8-1a33fa1981c8, owner=test}]> but
was: <[]>
at
org.apache.hadoop.ozone.recon.TestReconAsPassiveScm.testDatanodeRegistrationAndReports(TestReconAsPassiveScm.java:139)
{code}
Example run:
https://github.com/ss77892/ozone/actions/runs/41166900185/job/124448653275
*Root cause*
The test allocates a container in SCM, creates it on the datanode through
{{XceiverClientGrpc}}, and then checks right away that Recon has the same
containers as SCM:
{code:java}
runTestOzoneContainerViaDataNode(containerID, client);
// Verify Recon picked up the new container that was created.
assertEquals(scmContainerManager.getContainers(),
reconContainerManager.getContainers());
{code}
SCM knows about the container as soon as it allocates it. Recon finds out later
and asynchronously: the datanode sends an incremental container report to
Recon, and Recon then fetches the unknown container from SCM. Nothing in the
test waits for this. The assertion passes only if Recon processes the report
before the client calls in {{runTestOzoneContainerViaDataNode}} finish. On a
slow CI runner the report can arrive late, and Recon's container list is still
empty.
HDDS-16619 fixed the same kind of race in the pipeline check earlier in this
test, but the container check is still a one-shot assertion.
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
