andygrove opened a new pull request, #4868:
URL: https://github.com/apache/datafusion-comet/pull/4868
## Which issue does this PR close?
No dedicated issue. This addresses a repo-wide CI failure where native
builds fail at link time with:
```
error: linking with `cc` failed: exit status: 1
= note: /usr/bin/ld.bfd: cannot find -ljvm: No such file or directory
error: could not compile `datafusion-comet` (lib)
```
It has been hitting `PR Build (Linux)`, `Iceberg Spark SQL Tests`, and
`Spark SQL Tests` jobs across open PRs.
## Rationale for this change
The `hdfs-sys` dependency (pulled in by the default `hdfs-opendal` feature)
links `libjvm` and, from its build script, bakes an absolute `-L` search path
to the JDK's `libjvm` directory. Cargo caches that build-script output and
replays it when only downstream crates change, without re-running the build
script.
CI caches `native/target` keyed on `hashFiles('native/**/*.rs',
...Cargo.toml/lock)`. When a change does not touch native sources, the same
cache is restored. If the runner's JDK has moved in the meantime (`setup-java`
floats the Zulu patch version, so `JAVA_HOME`'s value can stay the same while
the underlying install is swapped), the replayed `-L` path points at a
directory that no longer exists, and the final link of `libcomet.so` fails with
`cannot find -ljvm`.
`core` is the crate that produces the final `cdylib`, so the fix belongs
there: emit a `-L` search path for the currently resolved JDK at build time.
The linker then finds `libjvm` regardless of any stale path replayed from a
dependency's cached build script.
## What changes are included in this PR?
- Add `native/core/build.rs` that:
- emits `cargo:rustc-link-search=native=$JAVA_HOME/lib/server` (the
`libjvm` location for every supported JDK, 11+), and
- emits `cargo:rerun-if-env-changed=JAVA_HOME` and
`cargo:rerun-if-changed=<that dir>` so the search path is refreshed
automatically whenever the JDK is swapped, rather than relying on a cache
invalidation.
No workflow or dependency changes are needed; the fix is self-contained in
the crate that performs the final link, so it applies to every native-build job.
## How are these changes tested?
Verbose local build (`cargo build -p datafusion-comet -vv`) confirms the
core build script now contributes the search path alongside the dependency's:
```
[hdfs-sys 0.3.0] cargo:rustc-link-search=native=<JDK>/lib/server
[datafusion-comet 1.0.0] cargo:rerun-if-changed=<JDK>/lib/server
[datafusion-comet 1.0.0] cargo:rustc-link-search=native=<JDK>/lib/server
```
so `-ljvm` resolves against the current JDK even when a dependency replays a
stale path. The existing native-build and Spark SQL CI jobs exercise the link
path that was failing.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
