FelixYBW commented on issue #12288:
URL: https://github.com/apache/gluten/issues/12288#issuecomment-6375617432
### Proposal: cache the Velox build in an image, refreshed when a Velox
update is merged
Most PRs don't change Velox, but every native build job compiles it from
scratch. On #13410 that was 93–166 min per job. On Velox bump PRs ccache barely
helps, because the cache key is `hashFiles('ep/build-velox/src/**')`.
**What the scripts already support**
- `builddeps-veloxbe.sh` accepts `--velox_home=<dir>` and can run single
steps. `... --velox_home=/opt/velox build_gluten_cpp` builds only Gluten C++
against an already-built Velox (`cpp/velox/CMakeLists.txt` links from
`${VELOX_HOME}/_build/release`).
- `get-velox.sh` reuses an existing `VELOX_HOME`: it resolves
`VELOX_REPO`/`VELOX_BRANCH` to a commit and only fetches and checks out when
the commit differs. If the image keeps the full `_build/release` tree, a PR
with a new Velox commit rebuilds incrementally.
**Design**
1. **Image per build variant.**
`apache/gluten:velox-static-almalinux-8-gcc13`, built `FROM
vcpkg-almalinux-8-gcc13`, runs `get-velox.sh` + `build-velox.sh` with exactly
the CI flags (`--enable_vcpkg=ON --enable_s3/gcs/hdfs/abfs=ON`, tests off) into
`/opt/velox`, and writes `/opt/velox/.velox-key`.
2. **Key** = hash of:
- the resolved Velox commit (`git ls-remote $VELOX_REPO $VELOX_BRANCH`,
or `VELOX_ENHANCED_BRANCH` for enhanced);
- `ep/build-velox/src/**`;
- `dev/vcpkg/**`;
- the variant's build flags.
A PR that changes `VELOX_REPO`, `VELOX_BRANCH` or `VELOX_ENHANCED_BRANCH`
in `get-velox.sh` gets a new key. So does a change to `build-velox.sh`, a
patch, or a vcpkg port, all of which change the Velox binaries without touching
`get-velox.sh`.
3. **Publish.** On push to `main` touching `ep/build-velox/**` or
`dev/vcpkg/**` (in practice every merged Velox update PR), `docker_image.yml`
builds and pushes `velox-static-<key>` plus a moving `velox-static-latest`. It
has to run on `apache/*` for the Docker Hub secrets.
4. **PR workflow.**
- A prep job computes the key and checks `docker manifest inspect
apache/gluten:velox-static-<key>`. The native build job uses `container: ${{
needs.prep.outputs.image }}`.
- **Image exists:** `builddeps-veloxbe.sh ... --velox_home=/opt/velox
build_gluten_cpp`, which builds Gluten C++ only.
- **No image** (the PR changes Velox or its build inputs, or the image
isn't published yet): fall back to today's base image and rebuild Velox at
workflow time.
- Tagging by key rather than only `latest` lets PRs based on an older
`main` still find their matching image.
**Variants**
| Variant | Used by | Velox build per PR today (#13410) |
|---|---|---|
| x86 static (`ci-velox-buildstatic-centos-8.sh`) | x86, Iceberg, Delta,
Kafka, nightly, ANSI | 93–118m each |
| x86 + ARM shared with test utils (`ci-velox-buildshared-centos-9.sh`) |
`cpp-test-udf-test` ×2 | 140m / 166m |
| ARM static (`ci-velox-buildstatic-centos-9.sh`) | ARM TPC | 133m |
| enhanced, cuDF | enhanced, cuDF | 111m / 76m |
Start with x86 static, then the two shared variants. Enhanced and cuDF are
probably not worth an image.
**To measure before committing**
- **Image size and runner disk.** A full Velox `_build/release` with
S3/GCS/HDFS/ABFS is likely many GB. If that is too large, strip the `.o` files
and keep the static libraries. That gives up the incremental rebuild for bump
PRs but keeps the exact-match fast path.
- **Gluten-only build time.** How long `build_gluten_cpp` takes on its own
with `NUM_THREADS=2` determines the real saving per job.
- **Bump PRs still build Velox.** The daily Velox bump PR has a new key by
definition, so it still builds Velox at workflow time. The saving is on all the
other PRs.
**Plan**
- [ ] PR 1: Dockerfile + `docker_image.yml` job, plus a key script shared by
the image build and the workflows (e.g. `dev/ci-velox-key.sh`)
- [ ] PR 2, after the image is published: prep job + `build_gluten_cpp` fast
path in `build-native-lib-centos-8`, then the other variants
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
