[
https://issues.apache.org/jira/browse/HDDS-17257?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
ASF GitHub Bot updated HDDS-17257:
----------------------------------
Labels: pull-request-available (was: )
> Intermittent failure in TestBlockDeletingService#testBlockDeletion
> ------------------------------------------------------------------
>
> Key: HDDS-17257
> URL: https://issues.apache.org/jira/browse/HDDS-17257
> Project: Apache Ozone
> Issue Type: Sub-task
> Components: test
> Reporter: Huang Kuan Hao
> Assignee: Huang Kuan Hao
> Priority: Major
> Labels: pull-request-available
>
> {code}
> [ERROR] Tests run: 73, Failures: 1, Errors: 0, Skipped: 4, Time elapsed:
> 27.20 s <<< FAILURE! -- in
> org.apache.hadoop.ozone.container.common.TestBlockDeletingService
> [ERROR]
> org.apache.hadoop.ozone.container.common.TestBlockDeletingService.testBlockDeletion(ContainerTestVersionInfo)[4]
> -- Time elapsed: 0.362 s <<< FAILURE!
> org.opentest4j.AssertionFailedError: Failed to read container checksum tree
> file:
> /tmp/junit-11171629283323802431/hdds/eca15dd6-87ff-4680-9d35-46d58ca44679/current/containerDir1/117454920117519356/metadata/117454920117519356.tree
> at org.junit.jupiter.api.AssertionUtils.fail(AssertionUtils.java:38)
> at org.junit.jupiter.api.Assertions.fail(Assertions.java:138)
> at
> org.apache.hadoop.ozone.container.common.TestBlockDeletingService.assertDeletionsInChecksumFile(TestBlockDeletingService.java:1311)
> at
> org.apache.hadoop.ozone.container.common.TestBlockDeletingService.lambda$testBlockDeletion$0(TestBlockDeletingService.java:628)
> at java.base/java.util.ArrayList.forEach(ArrayList.java:1604)
> at
> org.apache.hadoop.ozone.container.common.TestBlockDeletingService.testBlockDeletion(TestBlockDeletingService.java:628)
> {code}
> h3. Root cause
> {{deleteAndWait(service, n)}} waits until
> {{BlockDeletingServiceTestImpl#getTimesOfProcessed()}} reaches {{n}}. The
> counter is incremented when {{BackgroundService.PeriodicalTask#run()}}
> returns, but {{run()}} only submits each {{BlockDeletingTask}} with
> {{CompletableFuture.runAsync}} and does not wait for it, so the task may
> still be running when {{deleteAndWait}} returns.
> {{assertDeletionsInChecksumFile}} reads the container checksum file right
> after {{deleteAndWait}}, without waiting. If
> {{ContainerChecksumTreeManager#addDeletedBlocks}} has not written the file
> yet, the read fails as above. The other checks in {{testBlockDeletion}}
> already wait for the state they read, e.g. {{waitFor(() -> getBytesUsed() ==
> containerSpace / 3)}}. {{testChecksumFileUpdatedWhenDeleteRetried}} calls the
> same assertion right after {{deleteAndWait}}.
> A 500 ms sleep at the start of
> {{ContainerChecksumTreeManager#addDeletedBlocks}} reproduces it:
> {{testBlockDeletion}} and {{testChecksumFileUpdatedWhenDeleteRetried}} fail
> in all 8 parameterizations with the same error, and pass with the fix.
> h3. Fix
> Make {{assertDeletionsInChecksumFile}} wait until the checksum file records
> the expected number of deleted blocks.
> {{deleteAndWait}} could instead wait for the submitted tasks to finish (via
> {{BackgroundService#getFuture}}). That would also cover the success count
> check in {{testBlockDeletion}}, which is incremented a few statements after
> {{bytesUsed}}, but it changes the timing of every {{deleteAndWait}} caller,
> so it is left out here.
> Seen in:
> - https://github.com/apache/ozone/actions/runs/39010100374/job/117333028016
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
