andygrove opened a new issue, #1429: URL: https://github.com/apache/arrow-java/issues/1429
### Describe the bug, including details regarding any error messages, version, and platform. The macOS JNI library that the RC workflow builds (`jni-macos-aarch_64`) crashes the JVM the first time a dataset reads a JSON file. `TestFileSystemDataset#testBaseJsonRead` crashes every time on an Apple silicon Mac. The crash happens inside `new FileSystemDatasetFactory(...)`, while it reads the two-row JSON file the test writes: ``` # SIGSEGV (0xb) at pc=0x00000001644614e0, pid=57760, tid=63499 # Problematic frame: # C [jnilib-15442808047630463497.tmp+0x2c94e0] arrow::Result<simdjson::arm64::ondemand::object> arrow::internal::ResolveSimdjsonResult<simdjson::arm64::ondemand::object>(simdjson::simdjson_result<simdjson::arm64::ondemand::object>, std::__1::basic_string_view<char, std::__1::char_traits<char>>)+0x2c siginfo: si_signo: 11 (SIGSEGV), si_code: 2 (SEGV_ACCERR), si_addr: 0x0000000367f6a070 Native frames: C [jnilib-...tmp+0x2c94e0] arrow::internal::ResolveSimdjsonResult<simdjson::arm64::ondemand::object>(...)+0x2c C [jnilib-...tmp+0x1233a7c] arrow::json::(anonymous namespace)::ParseImpl::ParseValue(simdjson::arm64::ondemand::value&)+0x1400 C [jnilib-...tmp+0x12317d0] arrow::json::(anonymous namespace)::ParseImpl::Parse(std::__1::shared_ptr<arrow::Buffer> const&)+0x2cc C [jnilib-...tmp+0x123e978] arrow::json::(anonymous namespace)::ParseBlock(...)+0x408 C [jnilib-...tmp+0x123de54] arrow::json::(anonymous namespace)::DecodingOperator::operator()(...) const+0x48 ... C [jnilib-...tmp+0x12dad8c] std::__1::__thread_proxy<... arrow::internal::ThreadPool::LaunchWorkersUnlocked(int)::$_0>>(void*)+0x194 ``` The program counter offset is the same in every run. The fault address changes, but it always ends in `f6a070` (another run: `si_addr: 0x000000033af6a070`). **Why CI doesn't catch it** - Arrow C++ 26.0.0's JSON parser still uses RapidJSON. Arrow C++ `main` switched to simdjson in apache/arrow#51038. - The RC workflow builds the macOS library from Arrow C++ `main`. Except in the nightly, it builds the Linux libraries from the latest release. - The dataset tests run only in the Binaries job, on Linux x86_64. - So nothing tests the macOS library, which is the only one built from `main` on every run. `jni-macos` checks out `main` for every trigger, release tags included, so a release built now would ship this crash in the macOS `arrow-dataset` library. **What I ruled out** - Arrow C++ `main` on Linux. The nightly's Binaries job passes `TestFileSystemDataset` on x86_64 ([run 41256646402](https://github.com/apache/arrow-java/actions/runs/41256646402/job/124705109902)). That run's `jni-linux-aarch_64` library passes it on arm64 too: 18/18, twice, in an `eclipse-temurin:17-jdk` container on Apple silicon. So the simdjson parser works on arm64 Linux. - The memory allocator. The crash is identical with `ARROW_DEFAULT_MEMORY_POOL=system` and with `ARROW_DEFAULT_MEMORY_POOL=mimalloc`. **A lead I couldn't verify** The macOS runner already has Homebrew's simdjson installed: `brew update` lists `simdjson` among the outdated formulae. `ci/scripts/jni_macos_build.sh` builds the bundled simdjson 4.6.4 (`-Dsimdjson_SOURCE=BUNDLED`). Other Brewfile packages, such as Boost, Snappy, Thrift and xsimd, are installed under `/opt/homebrew`, so `/opt/homebrew/include` is probably on the include path. That raises a possibility: `arrow/json/parser.cc` could be compiled against Homebrew's `simdjson.h` but linked with the bundled simdjson, and a version mismatch like that could crash this way. The workflow already uninstalls Homebrew's RE2 and Protobuf because their files can get mixed into the build, and #1415 was a similar problem with lz4. Two ways to confirm or rule this out: - check which `simdjson.h` the build's `compile_commands.json` uses for `parser.cc`; - uninstall Homebrew's simdjson (`brew uninstall --ignore-dependencies simdjson`) before the build. **To reproduce** ``` gh run download 41256646402 -R apache/arrow-java -n jni-macos-aarch_64 tar -xzf jni-macos-aarch_64.tar.gz mvn -Parrow-jni -pl dataset -am test \ -Darrow.c.jni.dist.dir=$PWD/jni -Darrow.cpp.build.dir=$PWD/jni \ -Dtest='TestFileSystemDataset#testBaseJsonRead' -Dsurefire.failIfNoSpecifiedTests=false ``` The test JVM dies with `The forked VM terminated without properly saying goodbye`, and `dataset/hs_err_pid<pid>.log` has the stack above. Tested on: - macOS 26.6.2 on Apple silicon, Temurin 17.0.17. - arrow-java `main` (67ef5abf3). The crash also happens with the arrow-java changes from #1419. - Two macOS libraries: - from run 41233658912, built from Arrow C++ apache/arrow@b4a98b731d83: 5 of 5 runs crashed; - from the nightly run 41256646402, built from apache/arrow@d9a0ab4b088b: crashed. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
