[
https://issues.apache.org/jira/browse/HDDS-16423?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
ASF GitHub Bot updated HDDS-16423:
----------------------------------
Labels: pull-request-available (was: )
> Intermittent failure in
> TestContainerReconciliationWithMockDatanodes#testContainerReconciliationFailureContainerScan
> --------------------------------------------------------------------------------------------------------------------
>
> Key: HDDS-16423
> URL: https://issues.apache.org/jira/browse/HDDS-16423
> Project: Apache Ozone
> Issue Type: Sub-task
> Components: Ozone Datanode, test
> Reporter: Siyao Meng
> Assignee: Siyao Meng
> Priority: Major
> Labels: pull-request-available
>
> {code}
> [ERROR] Tests run: 16, Failures: 1, Errors: 0, Skipped: 0 <<< FAILURE! -- in
> org.apache.hadoop.ozone.container.keyvalue.TestContainerReconciliationWithMockDatanodes
> [ERROR]
> org.apache.hadoop.ozone.container.keyvalue.TestContainerReconciliationWithMockDatanodes.testContainerReconciliationFailureContainerScan
> -- Time elapsed: 0.005 s <<< FAILURE!
> org.opentest4j.AssertionFailedError: expected: <true> but was: <false>
> at org.junit.jupiter.api.Assertions.assertTrue(Assertions.java:183)
> at
> ...TestContainerReconciliationWithMockDatanodes$MockDatanode.scanContainer(TestContainerReconciliationWithMockDatanodes.java:561)
> at
> ...TestContainerReconciliationWithMockDatanodes.lambda$testContainerReconciliationFailureContainerScan$0(TestContainerReconciliationWithMockDatanodes.java:316)
> at java.base/java.util.ArrayList.forEach(ArrayList.java:1604)
> at
> ...TestContainerReconciliationWithMockDatanodes.testContainerReconciliationFailureContainerScan(TestContainerReconciliationWithMockDatanodes.java:316)
> {code}
> h3. Root cause
> The container replicas and the {{OnDemandContainerScanner}} are created once
> in {{@BeforeAll}} and shared by every test method. The
> {{MockDatanode.scanContainer}} helper asserts that
> {{OnDemandContainerScanner.scanContainerWithoutGap(...)}} returns a present
> {{Future}}:
> {code}
> Optional<Future<?>> scanFuture =
> onDemandScanner.scanContainerWithoutGap(containerSet.getContainer(containerID),
> TEST_SCAN);
> assertTrue(scanFuture.isPresent());
> {code}
> {{scanContainer}} returns an empty {{Optional}} when the container is already
> registered as in progress in {{containerRescheduleCheckSet}}; the entry is
> only removed by the scan executor task after the scan finishes.
> {{KeyValueHandler.reconcileContainer}} schedules a fire-and-forget scan via
> {{scanContainerWithoutGap}} in a {{finally}} block, so every reconciliation
> test leaves one scan in flight. Those tests wait via
> {{waitForExpectedScanCount(...)}}, which polls the {{numContainersScanned}}
> metric that is incremented inside the scan, before the executor thread runs
> {{removeContainerFromScheduledContainers}}. When the next test's synchronous
> {{scanContainer}} runs before that removal, the scan is not scheduled, the
> {{Optional}} is empty, and the assertion fails intermittently. The
> {{withoutGap}} helper uses {{minScanGap=0}}, so the scan-gap path cannot
> produce the empty {{Optional}}, and this test introduces no corruption, so
> the volume-failed path is not exercised; the only cause is the
> still-registered prior scan.
> h3. Fix
> In {{MockDatanode.scanContainer}}, wait until {{scanContainerWithoutGap}}
> returns a scheduled {{Future}} before blocking on it, instead of asserting
> the first call always schedules one. The single-thread scan executor is
> guaranteed to drain the prior scan, so this is deterministic by construction
> and keeps the per-call scan count at exactly one.
> - https://github.com/apache/ozone/actions/runs/34839737101/job/103965424574
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
