FrankChen021 opened a new issue, #19983: URL: https://github.com/apache/druid/issues/19983
<!-- Flaky Test template --> ## Summary `SchemalessTestSimpleTest.testFullOnTimeseries` and `SchemalessTestSimpleTest.testFullOnSearch` are flaky in the JDK 25 `S*` CI job. When they fail, they return empty results and the Surefire fork can subsequently terminate with a native `SIGSEGV`. This was observed while validating [PR #19981](https://github.com/apache/druid/pull/19981). The failure is in `org.apache.druid.segment.SchemalessTestSimpleTest`, not in a migrated test. ## CI evidence - [Original failed `test-jdk25-[S*]` job, workflow run 31571194115, attempt 1](https://github.com/apache/druid/actions/runs/31571194115/job/94033367389) - [Same job rerun, workflow run 31571194115, attempt 2](https://github.com/apache/druid/actions/runs/31571194115/job/94045315202) The original job reported two failures in this class and then a JVM crash. The rerun reproduced the same behavior: the Surefire retry mechanism ran the affected methods five times, and each method passed on runs 1, 2, 3, and 5 but failed on run 4. ## Exact test failures From the rerun log: ```text [ERROR] Tests run: 3, Failures: 2, Errors: 0, Skipped: 0 <<< FAILURE! -- in org.apache.druid.segment.SchemalessTestSimpleTest [ERROR] org.apache.druid.segment.SchemalessTestSimpleTest.testFullOnTimeseries org.opentest4j.AssertionFailedError: : key[rows] ==> expected: <11> but was: <0> at org.apache.druid.segment.SchemalessTestSimpleTest.testFullOnTimeseries(SchemalessTestSimpleTest.java:161) [ERROR] org.apache.druid.segment.SchemalessTestSimpleTest.testFullOnSearch org.opentest4j.AssertionFailedError: expected: <Result{timestamp=2011-01-12T00:00:00.000Z, value=SearchResultValue{value=[Hit{dimension='placementish', value='a'}, Hit{dimension='quality', value='automotive'}, Hit{dimension='placement', value='mezzanine'}, Hit{dimension='market', value='total_market'}]}}> but was: <Result{timestamp=2011-01-12T00:00:00.000Z, value=SearchResultValue{value=[]}}> at org.apache.druid.segment.SchemalessTestSimpleTest.testFullOnSearch(SchemalessTestSimpleTest.java:266) ``` The retry summary is: ```text [WARNING] Flakes: org.apache.druid.segment.SchemalessTestSimpleTest.testFullOnSearch Run 1: PASS Run 2: PASS Run 3: PASS Run 4: expected the four hits above, but got value=[] Run 5: PASS org.apache.druid.segment.SchemalessTestSimpleTest.testFullOnTimeseries Run 1: PASS Run 2: PASS Run 3: PASS Run 4: expected: <11> but was: <0> Run 5: PASS [WARNING] Tests run: 158, Failures: 0, Errors: 0, Skipped: 0, Flakes: 2 ``` This pass/fail/pass pattern confirms that both methods are flaky in this environment rather than deterministically broken. ## JVM crash evidence The same rerun log contains: ```text A fatal error has been detected by the Java Runtime Environment: SIGSEGV (0xb) at pc=0x00007fcd8c227851, pid=3073, tid=3075 JRE version: OpenJDK Runtime Environment Zulu25.36+15-CA (25.0.4+7) Java VM: OpenJDK 64-Bit Server VM Zulu25.36+15-CA (25.0.4+7-LTS, mixed mode, sharing, tiered, compressed oops, compressed class ptrs, g1 gc, linux-amd64) An error report file with more information is saved as: /home/runner/work/druid/druid/processing/hs_err_pid3073.log The forked VM terminated without properly saying goodbye. VM crash or System.exit called? Process Exit Code: 134 Crashed tests: org.apache.druid.segment.SchemalessTestSimpleTest ``` The fork command also included `-Xmx2048m`, `-XX:MaxDirectMemorySize=2500m`, `--add-modules=jdk.incubator.vector`, and JaCoCo; the CI log identifies the runtime as Zulu/OpenJDK 25.0.4 on `linux-amd64`. ## Expected behavior Both tests should be stable and the JDK 25 job should complete without a native JVM crash. ## Investigation status The evidence establishes: 1. Both test methods intermittently produce empty query/segment results. 2. The same failure reproduced in two CI attempts. 3. The failure is specific to the JDK 25 CI environment observed here and is followed by a native JVM crash. 4. The hosted log reports that `hs_err_pid3073.log` was written, but the file is not exposed in the job output, so the native crash stack is not available for subsystem-level attribution. A direct-buffer, vector, or other JDK/runtime interaction is a hypothesis based on the JVM configuration, not a confirmed root cause. ## Suggested follow-up - Preserve and upload the `hs_err_pid*.log` file as a CI artifact when a JVM crash occurs. - Compare this test on the JDK versions used by the matrix and on another JDK 25 distribution/build. - Isolate the test with the vector module, JaCoCo, and direct-memory settings varied independently. - Consider quarantining or retrying these tests in the JDK 25 job until the runtime issue is understood. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
