Siyao Meng created HDDS-17346:
---------------------------------
Summary: Intermittent timeout in container scanner integration
tests waiting for container close
Key: HDDS-17346
URL: https://issues.apache.org/jira/browse/HDDS-17346
Project: Apache Ozone
Issue Type: Sub-task
Components: Ozone Datanode, test
Reporter: Siyao Meng
Assignee: Siyao Meng
{code}
[ERROR]
org.apache.hadoop.ozone.dn.scanner.TestBackgroundContainerMetadataScannerIntegration.testCorruptionDetected(ContainerTestCorruptions)[2]
-- Time elapsed: 5.119 s <<< ERROR!
java.util.concurrent.TimeoutException:
Timed out waiting for condition. Thread diagnostics:
...
at
org.apache.ozone.test.GenericTestUtils.waitFor(GenericTestUtils.java:137)
at
org.apache.hadoop.ozone.dn.scanner.ContainerScannerIntegrationTests.closeContainerAndWait(ContainerScannerIntegrationTests.java:169)
at
org.apache.hadoop.ozone.dn.scanner.ContainerScannerIntegrationTests.writeDataThenCloseContainer(ContainerScannerIntegrationTests.java:162)
at
org.apache.hadoop.ozone.dn.scanner.ContainerScannerIntegrationTests.writeDataThenCloseContainer(ContainerScannerIntegrationTests.java:151)
at
org.apache.hadoop.ozone.dn.scanner.TestBackgroundContainerMetadataScannerIntegration.testCorruptionDetected(TestBackgroundContainerMetadataScannerIntegration.java:96)
{code}
h3. Root cause
{{ContainerScannerIntegrationTests#closeContainerAndWait}} asks SCM to close
the container, then gives the datanode only 5 seconds to report it CLOSED. The
close is applied through Ratis, and {{KeyValueContainer#close}} calls
{{flushAndSyncDB}}, which fsyncs the RocksDB WAL. In the failed run SCM
triggered the close of container #3 at 11:32:56.5. At the timeout (11:33:01.5)
the thread dump shows the {{ContainerOp}} thread still in {{RocksDB.flushWal}}
under {{KeyValueContainer.close}}. The container was closed at 11:33:03.0,
about 6.5 seconds after the close request. The first case's close in the same
run already took about 3 seconds, so 5 seconds is not enough headroom on a slow
CI disk.
h3. Fix
Raise the timeout of the datanode close wait in {{closeContainerAndWait}} from
5 to 30 seconds. The wait polls, so passing runs finish just as fast. The
helper is shared by all container scanner integration tests.
- https://github.com/apache/ozone/actions/runs/39763337594/job/119785408645
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
