LuciferYang opened a new pull request, #12903:
URL: https://github.com/apache/gluten/pull/12903
## What changes are proposed in this pull request?
`dev/docker/ubuntu/Dockerfile.ubuntu22-m2-cache` pre-populates the Maven
cache with one `dependency:go-offline` per supported Spark version. The stanza
commented `# Spark 4.0` passes `-Pspark-4`, which is not a profile id: the root
pom defines `spark-3.4`, `spark-3.5`, `spark-4.0` and `spark-4.1`. Maven prints
`The requested profile "spark-4" could not be activated because it does not
exist.`, continues with no Spark profile active, and that stanza resolves
against the root pom's default properties instead of Spark 4.0's.
`enforce-spark-profile` does not catch it. `dependency:go-offline` invokes
the plugin goal directly rather than running a lifecycle, and that enforcer
execution binds to `validate`, so it never fires. The command exits 0 after the
warning, which is why this went unnoticed since #11655.
The defaults lean 3.5, so the stanza caches a mix:
| property | `-Pspark-4` (no profile active) | `-Pspark-4.0` |
|-|-|-|
| `spark.version` | 4.0.2 | 4.0.2 |
| `delta.version` | 3.3.2 | 4.0.1 |
| `sparkshim.artifactId` | `spark-sql-columnar-shims-spark35` |
`spark-sql-columnar-shims-spark40` |
| `sparkbundle.version` | 3.5 | 4.0 |
`spark.version` agrees by accident, because the root default is already
4.0.2, so the Spark 4.0 jars do land in the image. Delta 4.0.1 and the spark40
shim coordinates do not, and the Spark 4.0 jobs fetch them over the network,
which is the cost this image exists to remove. After the fix that stanza does
real work, so the image build gets slower and the image gets larger by whatever
Spark 4.0's Delta and shim dependencies weigh. That is the intent.
Split out of #12902, which removes Spark 3.3 support and had to edit the
same `RUN` for an unrelated reason (deleting the trailing Spark 3.3 stanza left
a dangling `; \`). This change adds behavior rather than removing it, so it is
easier to review and to revert on its own.
## How was this patch tested?
The property table is reproducible from the repo root, and it is the whole
substance of the change:
```
for P in spark-4 spark-4.0; do
for E in spark.version delta.version sparkshim.artifactId
sparkbundle.version; do
./build/mvn -q -N
-Pjava-17,scala-2.13,$P,backends-velox,hadoop-3.3,spark-ut \
help:evaluate -Dexpression=$E -DforceStdout
done
done
```
`docker buildx build --check` on the file reports the same single
pre-existing `NoEmptyContinuation` warning before and after, so the edit
changes nothing structural.
The image is built by `docker_image.yml`, which triggers on push to main
under `dev/docker/ubuntu/*` and on a weekly cron, so the effect appears when
that workflow runs rather than in PR CI.
## Was this patch authored or co-authored using generative AI tooling?
Generated-by: Claude claude-opus-5
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
