Siyao Meng created HDDS-16423:
---------------------------------
Summary: Intermittent failure in
TestContainerReconciliationWithMockDatanodes#testContainerReconciliationFailureContainerScan
Key: HDDS-16423
URL: https://issues.apache.org/jira/browse/HDDS-16423
Project: Apache Ozone
Issue Type: Sub-task
Components: Ozone Datanode, test
Reporter: Siyao Meng
Assignee: Siyao Meng
{code}
[ERROR] Tests run: 16, Failures: 1, Errors: 0, Skipped: 0 <<< FAILURE! -- in
org.apache.hadoop.ozone.container.keyvalue.TestContainerReconciliationWithMockDatanodes
[ERROR]
org.apache.hadoop.ozone.container.keyvalue.TestContainerReconciliationWithMockDatanodes.testContainerReconciliationFailureContainerScan
-- Time elapsed: 0.005 s <<< FAILURE!
org.opentest4j.AssertionFailedError: expected: <true> but was: <false>
at org.junit.jupiter.api.Assertions.assertTrue(Assertions.java:183)
at
...TestContainerReconciliationWithMockDatanodes$MockDatanode.scanContainer(TestContainerReconciliationWithMockDatanodes.java:561)
at
...TestContainerReconciliationWithMockDatanodes.lambda$testContainerReconciliationFailureContainerScan$0(TestContainerReconciliationWithMockDatanodes.java:316)
at java.base/java.util.ArrayList.forEach(ArrayList.java:1604)
at
...TestContainerReconciliationWithMockDatanodes.testContainerReconciliationFailureContainerScan(TestContainerReconciliationWithMockDatanodes.java:316)
{code}
h3. Root cause
The container replicas and the {{OnDemandContainerScanner}} are created once in
{{@BeforeAll}} and shared by every test method. The
{{MockDatanode.scanContainer}} helper asserts that
{{OnDemandContainerScanner.scanContainerWithoutGap(...)}} returns a present
{{Future}}:
{code}
Optional<Future<?>> scanFuture =
onDemandScanner.scanContainerWithoutGap(containerSet.getContainer(containerID),
TEST_SCAN);
assertTrue(scanFuture.isPresent());
{code}
{{scanContainer}} returns an empty {{Optional}} when the container is already
registered as in progress in {{containerRescheduleCheckSet}}; the entry is only
removed by the scan executor task after the scan finishes.
{{KeyValueHandler.reconcileContainer}} schedules a fire-and-forget scan via
{{scanContainerWithoutGap}} in a {{finally}} block, so every reconciliation
test leaves one scan in flight. Those tests wait via
{{waitForExpectedScanCount(...)}}, which polls the {{numContainersScanned}}
metric that is incremented inside the scan, before the executor thread runs
{{removeContainerFromScheduledContainers}}. When the next test's synchronous
{{scanContainer}} runs before that removal, the scan is not scheduled, the
{{Optional}} is empty, and the assertion fails intermittently. The
{{withoutGap}} helper uses {{minScanGap=0}}, so the scan-gap path cannot
produce the empty {{Optional}}, and this test introduces no corruption, so the
volume-failed path is not exercised; the only cause is the still-registered
prior scan.
h3. Fix
In {{MockDatanode.scanContainer}}, wait until {{scanContainerWithoutGap}}
returns a scheduled {{Future}} before blocking on it, instead of asserting the
first call always schedules one. The single-thread scan executor is guaranteed
to drain the prior scan, so this is deterministic by construction and keeps the
per-call scan count at exactly one.
- https://github.com/apache/ozone/actions/runs/34839737101/job/103965424574
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
