FelixYBW opened a new pull request, #13422:
URL: https://github.com/apache/gluten/pull/13422
## What changes are proposed in this pull request?
Part of #12288. On a typical PR (#13410), 59 of 80 CI jobs installed
packages at runtime, mostly on top of the pre-built `apache/gluten` images: JDK
17, Python + pyspark/pandas/pyarrow, gh/jq/bison, clang-tools-extra.
Package-mirror errors from those installs caused 3 of that PR's 4 failures,
each needing a rerun. This PR bakes those tools into the images. A follow-up PR
(opened as a draft) switches the workflows to them once the images are
published.
**Build images**
- **`dev/docker/install-ci-tools.sh`** (new): installs gh 2.63.2 and jq
1.7.1, the same versions the jobs download today. With
`--pyspark-python=<python>`, it also creates one virtualenv per PySpark version
used by the Spark UTs: `/opt/pyspark/3.4.4` and `/opt/pyspark/3.5.5`, each with
setuptools 77.0.3, cython, pandas 2.2.3, pyarrow 20.0.0, regex and requests.
Jobs select one by putting its `bin/` first in `PATH`, so the system `python3`
(and dnf) is untouched.
- **`centos-8-jdk{8,11,17}` / `centos-9-jdk{8,11,17}`**: add JDK 17 next to
the default JDK, plus procps-ng, which, findutils, gzip, tzdata and the PySpark
venvs.
- CentOS 8 also gets patchelf, with venvs on python3.9.
- CentOS 9 also gets `regex`/`requests` for `dev/check.py`, with venvs on
python3.11.
- **`vcpkg-almalinux-8-gcc13`, `vcpkg-centos-9`**: add gh, jq, tzdata and
python3.9. This replaces the "Install CI tools (temporary - until image is
rebuilt …)" steps.
**New lean runtime images for the TPC jobs**
The TPC jobs exist to check that the static native library runs on a plain
OS. Moving them onto the heavy build images, which ship Arrow and other
libraries, could hide missing dependencies. So these images contain only the
OS, JDKs and CI tools:
| Image | Base | Adds |
|---|---|---|
| `centos-8-runtime` | `centos:8` | JDK 8/11/17, Celeborn 0.5.4/0.6.3
releases |
| `centos-9-runtime` (x86 + arm64) | `quay.io/centos/centos:stream9` | JDK
8/11/17/21 |
| `ubuntu-22.04-runtime` | `ubuntu:22.04` | JDK 8/11/17/21/25, Hadoop 3.3.6,
MinIO |
`install-spark-deps.sh` now reuses a pre-installed `/opt/hadoop-<version>`
and MinIO instead of downloading them.
**Also**
- `docker_image.yml` publishes the three runtime images.
- `dev/docker/Makefile` now builds from the repo root so `COPY
dev/docker/...` resolves.
- Updated `velox-backend-CI.md`.
## How was this patch tested?
- The images can only be pushed from `apache/*`, so I ran a temporary smoke
workflow on my fork. It built the new layers on top of the published images and
built the runtime images, all 13 jobs green:
- **Build images:** `centos-8-jdk8`, `centos-8-jdk17`, `centos-9-jdk8`,
`vcpkg-almalinux-8-gcc13` on x86, and `centos-9-jdk17`, `vcpkg-centos-9` on
arm64. Checked gh/jq, JDK 17, and `import pyspark, pandas, pyarrow, regex` in
both venvs (3.4.4 / 3.5.5 with pandas 2.2.3 / pyarrow 20.0.0).
- **Runtime images:** `centos-8-runtime`, `centos-9-runtime` on both x86
and arm64, and `ubuntu-22.04-runtime`. Checked every JDK's `java -version`,
gh/jq/ps, the Celeborn tarballs, and `install_hadoop` + `setup_hdfs` +
`install_minio` using the pre-installed copies.
- **Sizes (uncompressed):** `centos-8-runtime` 1.9 GB, `centos-9-runtime`
1.8 GB, `ubuntu-22.04-runtime` 3.4 GB.
- `actionlint` on `docker_image.yml`: clean.
- License header check: clean.
## Was this patch authored or co-authored using generative AI tooling?
Generated-by: Claude claude-opus-5-5
🤖 Generated with [Claude Code](https://claude.com/claude-code)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
