[
https://issues.apache.org/jira/browse/HDDS-16750?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=18129447#comment-18129447
]
Siyao Meng commented on HDDS-16750:
-----------------------------------
Recurred again on a PR build:
https://github.com/apache/ozone/actions/runs/39542474581/job/119042161642
(apache/ozone#11619, commit 651fa9706b5).
All Recon test classes reported 0 failures and 0 errors, but ozone-recon failed
with the same SurefireBooterForkException at
ForkStarter.fork(ForkStarter.java:577). This time the native abort came right
after TestFeaturesEndPoint finished:
{code}
[INFO] Tests run: 4, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 4.865 s
-- in org.apache.hadoop.ozone.recon.api.TestFeaturesEndPoint
terminate called after throwing an instance of 'std::bad_alloc'
what(): std::bad_alloc
Aborted (core dumped)
{code}
The jvmRun1 dump again only has the PpidChecker "Stream closed" / "error
[STOPPED] to read process" entries.
Since reuseForks=false, the abort seems to happen when the forked JVM exits
after its test class completes. The class differs per run
(TestNSSummaryDiskUsageOrdering, TestReconStorageContainerManagerFacade, now
TestFeaturesEndPoint), but all three open an OM metadata manager (RocksDB)
through OMMetadataManagerTestUtils.initializeNewOmMetadataManager /
getTestReconOmMetadataManager and never close it. This is common in the module:
20 of the 24 Recon test classes using those helpers have no close() or stop()
call. Live RocksDB handles at JVM exit racing native static destructors could
explain the abort. Not confirmed (no core dump), but closing these DBs in
teardown may be worth trying.
> Investigate Surefire fork failure in Recon CI with no reported test failures
> ----------------------------------------------------------------------------
>
> Key: HDDS-16750
> URL: https://issues.apache.org/jira/browse/HDDS-16750
> Project: Apache Ozone
> Issue Type: Bug
> Reporter: Huang Kuan Hao
> Assignee: Huang Kuan Hao
> Priority: Major
>
> The integration (recon) check failed on b41d78c5da (upstream base 402a0622fe)
> with JDK 25.0.4.1 and Surefire 3.5.2. Recon reported 790 unit tests (0
> failures, 0 errors, 32 skipped) and 39 integration tests (0 failures, 0
> errors), but Maven failed with a Surefire fork error.
> {code}
> ExecutionException Error occurred in starting fork, check output in log
> SurefireBooterForkException at ForkStarter.fork(ForkStarter.java:577)
> {code}
> The integration-recon artifact contains a JVM dump with PpidChecker,
> IOException: Stream closed, and IllegalStateException: error [STOPPED] to
> read process 2652. These are observed symptoms; their connection to the
> unsuccessful fork exit is not yet established. Investigate the fork failure
> and identify the affected test or process.
> HDDS-13307 addressed log scanning after successful test retries; this run
> reports an actual Surefire fork failure.
> CI:
> https://github.com/rich7420/ozone/actions/runs/37595249924/job/112711390698
> Artifact:
> https://github.com/rich7420/ozone/actions/runs/37595249924/artifacts/11472896153
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
