baibaichen opened a new pull request, #13362:
URL: https://github.com/apache/gluten/pull/13362
## What changes are proposed in this pull request?
Build Gluten and the prepared Velox source tree in one CMake graph, replacing
imports of prebuilt Velox archives and manually reconstructed dependencies
with real upstream targets.
### Benefits
**For developers: a unified IDE experience.**
Gluten, Velox, tests, and benchmarks are visible in the same CMake project.
Developers can navigate across both codebases, select build targets, and
debug
without switching between separately configured projects.
**For AI coding agents: a more reliable edit–build–verify loop.**
An explicit target graph reduces the build knowledge agents must reconstruct
from shell scripts and prebuilt artifact paths. Building the JNI backend also
builds its prerequisites, and relevant Gluten and Velox test targets can be
selected from the same project. This reduces the risk of missing a rebuild
or validating changes against stale native libraries.
### Implementation
- Integrate Velox using `add_subdirectory()` and its actual dependency
targets.
- Preserve the existing object-library/JNI separation and the `libgluten.so`
and `libvelox.so` output names.
- Distinguish the upstream `velox` target from `gluten_velox_backend`, with
compatibility aliases for existing consumers.
- Build the unified graph once in the default flow; request the JNI backend
in the incremental entry.
- Forward Velox test, benchmark, and CUDA settings; update GPU artifact
paths.
- Retain the main script's clean-build behavior and existing dependency
installation options.
The main CMake project continues to require a prepared `VELOX_HOME`; it does
not automatically acquire sources. This changes build orchestration, not the
execution architecture, and makes no query-performance claim.
## How was this patch tested?
- Local Linux/vcpkg native compilation and Spark 4.1 Maven packaging
completed
successfully after rebasing onto main.
- Eight isolated build-script regression tests passed. These use stubs rather
than real dependency installation, native compilation, or Maven
invocations.
- Shell syntax, license-header, and diff checks passed.
Native test and benchmark binaries were not executed during local validation.
This PR is a draft pending broader platform validation.
### CI validation to watch
- [ ] Non-vcpkg Linux and ARM: dependency discovery, native linking, and C++
tests.
- [ ] macOS: Folly/glog/gflags linkage and JNI startup, especially duplicate
flag registration. The weekly job exercises TPC-H, not a full CTest
run.
- [ ] GPU/cuDF: CUDA parameters, compatibility aliases, dependency paths, and
test compilation with the existing experimental memory-resource macro
scope. The GPU job does not execute GPU tests.
Additional scenarios not established by passing the default PR jobs:
- Arrow available only in the old bundled installation location.
- GCC 13 Release with Velox's own tests enabled, including the decoder-test
warning workaround.
- `ENABLE_JEMALLOC_STATS=ON`.
- Missing GTest with dependency resolution set to `SYSTEM`.
## Was this patch authored or co-authored using generative AI tooling?
Generated-by: Claude Code 2.1.303
🤖 Generated with [Claude Code](https://claude.com/claude-code)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
