dfa1 opened a new issue, #3459: URL: https://github.com/apache/maven-surefire/issues/3459
## Affected version 3.6.0 (regression). Bisected to commit [90529e9](https://github.com/apache/maven-surefire/commit/90529e9fab1a2df720d9a88c1abb6c34a4ab5336) ("Use StackWalker in StackTraceProvider when available (Java 9+)", #3374). Last known good: 3.6.0-M1 and everything before that commit. ## Environment - OS: windows-latest GitHub Actions runner (Windows Server 2022/2025) - JDK: Zulu 25.0.4-7, x64 - Build: Maven, `reuseForks=false`, `argLine` includes `--enable-native-access=ALL-UNNAMED` - Project: https://github.com/dfa1/rocksdb-ffm (Java FFM bindings for RocksDB's C API) ## Bug description Bumping `maven-surefire-plugin` from 3.5.6 to 3.6.0 makes three specific test classes deterministically kill their forked JVM on `windows-latest` only. macOS and Linux (x86_64 and aarch64) are unaffected. The same commit set with surefire pinned at 3.5.6 (or at 3.6.0-M1) passes on Windows every time. ``` [ERROR] ExecutionException The forked VM terminated without properly saying goodbye. VM crash or System.exit called? [ERROR] Crashed tests: io.github.dfa1.rocksdbffm.CompactionFilterFactoryTest io.github.dfa1.rocksdbffm.CompactionFilterTest io.github.dfa1.rocksdbffm.EventNotifierTest ``` This reproduced identically across many independent CI runs — always exactly these three classes, never a different subset. ### What we ruled out - **Not a classic native crash**: no `hs_err_pid*.log` is ever produced, despite surefire's own message suggesting a VM crash. If HotSpot's signal handler had caught a fault normally, that file would exist. - **Not the "Boot Manifest-JAR" classpath warning**: every single fork (150+, since `reuseForks=false`) logs `Boot Manifest-JAR contains absolute paths in classpath '...', 'other' has different root` to the `.dumpstream` file (a `Path.relativize()` failure across Windows drive roots, falling back to absolute paths in the manifest jar). This happens for both passing and crashing forks alike — harmless noise, unrelated to the real bug. - **Not simply "any test with a native upcall crashes"**: other tests in the same suite that register FFM upcalls but only ever invoke them synchronously on the JVM's own calling thread (e.g. a custom merge operator callback, or a test that calls `System.exit` from a JVM-owned thread) pass without issue. ### Bisection We built surefire from source at each candidate commit between `surefire-3.6.0-M1` and `surefire-3.6.0` (54 commits) and ran our suite against each, narrowing by binary search: | Round | Commit (index in the 54) | Result | |---|---|---| | baseline | 3.6.0-M1 | pass | | 1 | `37f0725` (26) | crash | | 2 | `381c404` (12) | crash | | 3 | `c633042` (5) | pass | | 4 | `90529e9` (8) | **crash** | | 5 | `e759c64` (6) | pass | | 6 (final) | `523e1ce` (7) | pass | Index 7 passing and index 8 crashing pins the boundary exactly: commit `90529e9` is the first bad commit in the range. The regression is isolated to that single commit, [90529e9](https://github.com/apache/maven-surefire/commit/90529e9fab1a2df720d9a88c1abb6c34a4ab5336) / PR [#3374](https://github.com/apache/maven-surefire/pull/3374), which switches `StackTraceProvider.getStack()` from `Thread.currentThread().getStackTrace()` to `java.lang.StackWalker.getInstance(options, depth).walk(...)` (falling back to the old path only if the `StackWalker` call throws). ### Suspected mechanism `CompactionFilter`/`CompactionFilterFactory`/`EventNotifier` register a native → Java callback that RocksDB invokes from its own background threads (the compaction thread pool, and flush/compaction event-listener threads) — not threads the JVM created. The callback path is a JDK 25 `Linker.upcallStub()` entry point, which the JVM implements as a generated hidden class/`MethodHandle` trampoline. The first time such a background thread fires the upcall, the JVM attaches that foreign thread, and from the JVM's point of view its call stack starts at that synthetic hidden-class frame rather than a normal `Thread.run()`-rooted chain. `Thread.getStackTrace()` (still used as the fallback in the patched code) tolerates that stack shape fine — this project's whole suite passed under every surefire version before this commit. `StackWalker`, by contrast, does extra classification work on each frame specifically to hide reflection/lambda frames (per the PR's own code comment), which means it's doing more invasive introspection of frame metadata for exactly the kind of synthetic/hidden frame an FFM upcall trampoline produces. We suspect that introspection is hitting a rough edge in HotSpot's stack-walking internals for hidden-class/MethodHandle frames belonging to a JVM-attached (not JVM-created) thread, causing a hard, in-process JVM fault severe enough that the normal crash-dump path (`hs_err_pid*.log`) never runs. We have not attached a native debugger to confirm this at the HotSpot source level — it's our best explanation grounded in the diff, not a proven root cause. ### Workaround Setting `-Dsurefire.stackTraceMaxFrames=0` (or `<stackTraceMaxFrames>0</stackTraceMaxFrames>` in the plugin `<configuration>`) should avoid the crash entirely, based on reading the source: `StackTraceProvider.getStack()` returns `Collections.emptyList()` before ever reaching `StackWalkerStrategy`, so `StackWalker` is never invoked. This would let a consumer stay on 3.6.0 rather than pinning back to 3.5.6, at the documented cost of losing the console-output-to-test-class association this option controls — not a real fix, but a plausible interim mitigation. ## Reproduction Not yet minimized to a surefire-only reproducer (would require a small native library exercising a JDK 25 `Linker.upcallStub` invoked from a non-JVM-created thread, run under `StackTraceProvider.getStack()`'s new path). Happy to attempt one if useful — our current reproduction is the full `dfa1/rocksdb-ffm` CI matrix. ## What we'd like from surefire maintainers - Whether `StackWalker.getInstance(...).walk(...)` in `StackWalkerStrategy` is known to interact badly with frames belonging to a foreign/attached (not JVM-created) thread, particularly hidden-class or `MethodHandle` frames such as an FFM upcall trampoline. - Whether a dedicated opt-out (independent of `stackTraceMaxFrames`, which also disables the underlying feature) is worth adding, so a consumer can keep stack-trace-to-console-output association while forcing the legacy `Thread.getStackTrace()` path. As it stands, `StackWalkerStrategy.walk`'s `catch (ReflectiveOperationException | RuntimeException)` can't help here since a native JVM fault isn't a Java exception at all. Happy to run additional diagnostics on our CI if that helps narrow this down further. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
