zhangfengcdt opened a new pull request, #6493: URL: https://github.com/apache/datafusion-comet/pull/6493
## Which issue does this PR close? Closes #6096. ## Rationale for this change The cause is that the native library gets loaded more than once. `CometSparkSessionExtensionsSuite` resets `NativeBase`'s `loaded` flag and calls `load()` again. Each call unpacks the bundled library to a new, randomly named temporary file and loads it. To the JVM and the OS loader that is a separate library with its own statics, so the process ends up with several copies and only the first has `JAVA_VM` set. JDK 17 resolves a JNI method on first call by iterating a hash map of loaded libraries, so a method first called after that suite can bind to an uninitialized copy. Which copy wins depends on the random file names, which is why the failure is intermittent. `setShufflePartitionPusher` is first called in `CometNativeShuffleSuite`, which is why the abort lands there. ## What changes are included in this PR? - `NativeBase`: a flag that is never reset records that the bundled library has been loaded, so a repeat `load()` skips unpacking and loading another copy. The rest of the load sequence is unchanged. - `CometSparkSessionExtensionsSuite`: a regression test that resets the flag, reloads, and asserts no new library file was unpacked. ## How are these changes tested? The new regression test fails without the fix and passes with it. I also reproduced the issue by running `CometSparkSessionExtensionsSuite` followed by `CometNativeShuffleSuite` in one JVM, on macOS with Spark 4.1. - Without the fix: 4 of 5 runs failed with `JAVA_VM not initialized`, starting at the test right after the callback registration test, as in the issue. - With the fix: 9 of 9 runs passed, with exactly one library copy unpacked. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
