pingzh opened a new pull request, #5511: URL: https://github.com/apache/datafusion-comet/pull/5511
## Which issue does this PR close? Part 1 of #5490. This PR does not close the issue; image publication and CI adoption will follow separately. ## Rationale for this change Linux Spark CI jobs repeatedly install toolchains and download the same dependencies. Before switching those jobs to a prebuilt image, we need a pinned toolchain recipe and a check that its dependency caches are sufficient without carrying forward Comet binaries or previous build outputs. ## What changes are included in this PR? - Add a Linux/amd64 pilot for Spark 4.1.3 and Zulu JDK 17, with pinned base-image digests, tool versions, Spark sources, and SBT launcher checksums. - Build Comet and Spark in an intermediate stage, then export third-party Cargo, Maven, Coursier, SBT, and Ivy caches under `/opt/comet-ci`. The final image starts from the toolchain stage and excludes Comet artifacts, SNAPSHOTs, local Maven installations, credentials, and workspace build outputs. - Add a fresh-source validator that disables networking, mounts an empty runtime home and workspace, checks dependency fingerprints, rebuilds native and JVM Comet, compiles Catalyst/SQL/Hive tests, and runs two Spark smoke suites. - Document the build and cache contract, and add dependency-free regression tests. Existing CI workflows remain unchanged. Publishing, scheduled refreshes, digest-based adoption, rollback, and performance measurements are outside this PR. ## How are these changes tested? - `python3 dev/ci/images/test_image.py`: 21 tests passed. - Complete image build passed, including native compilation, the full Maven reactor (`-DskipTests`, with test classes compiled), Catalyst/SQL/Hive test compilation, and 104 Spark smoke tests: 42 in `LiteralExpressionSuite` and 62 in `MathFunctionsSuite`. - `COMET_CI_BUILD_JOBS=16 bash dev/ci/images/validate.sh`: passed the same builds and all 104 smoke tests in a fresh container with `--network=none`, an empty runtime home, and no host dependency-cache mounts. - Toolchain checks passed with networking disabled. ShellCheck, Ruff, Prettier, and Maven RAT checks passed. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
