voonhous opened a new pull request, #19888:
URL: https://github.com/apache/hudi/pull/19888
### Describe the issue this Pull Request addresses
Closes #19887. Follow-up to #19884.
The docker demo stack and the docker-cluster ITs are pinned to Spark 3.5.3
while the root pom moves to 3.5.9 in #19884. That leaves the containers running
a Hudi bundle compiled against a newer Spark than their runtime, the direction
that can hit a missing API.
### Summary and Changelog
Every 3.5.3 pin in the docker tree moves to 3.5.9 in one go. 15 files, 48
lines, no logic changes:
-
`docker/hoodie/hadoop/{spark_base,sparkadhoc,sparkmaster,sparkworker}/Dockerfile`
`ARG SPARK_VERSION`, `docker/hoodie/hadoop/pom.xml` `docker.spark.version`,
the `build_docker_images.sh` default
- both compose sets renamed `spark353` -> `spark359`, with their `image:`
tags and `CLUSTER_NAME`
- `setup_demo.sh` / `stop_demo.sh`, `hudi-integ-test/pom.xml`
`dockerCompose.file`, `docker/README.md`, and the parked
`integration-tests-hive-sync` matrix in `bot.yml`
**Do not merge until the 3.5.9 images are on Docker Hub.** The
`integration-tests` job runs `docker compose up` on
`docker-compose_hadoop284_hive2310_spark359_amd64.yml` on every run, so until
the images exist that job fails on the pull. That red check is the gate, not a
defect in the PR.
<details>
<summary>For whoever has push rights on the apachehudi org: build and
push</summary>
From this branch, after `mvn -Pintegration-tests clean package -DskipTests`
so the bundles land in the docker `target/` dirs:
```shell
cd docker
# hadoop 2.8.4 / hive 2.3.10 / spark 3.5.9 (the script defaults)
./build_docker_images.sh --multi-arch
# hadoop 3.3.4 / hive 3.1.3 / spark 3.5.9 (what setup_demo.sh uses)
./build_docker_images.sh --hadoop-version 3.3.4 --hive-version 3.1.3
--spark-version 3.5.9 --multi-arch
```
`--multi-arch` builds amd64 and arm64 and pushes in one pass; it needs a
`docker buildx` builder and a `docker login` with push rights (steps in
`docker/README.md`). Eight new repositories:
-
`apachehudi/hudi-hadoop_2.8.4-hive_2.3.10-spark{base,adhoc,master,worker}_3.5.9`
-
`apachehudi/hudi-hadoop_3.3.4-hive_3.1.3-spark{base,adhoc,master,worker}_3.5.9`
The Spark version is part of the repository name and the compose files pull
`:latest`, so this creates new repos and touches nothing currently pinned. Once
pushed, re-run the checks on this PR.
</details>
<details>
<summary>Verification</summary>
- `docker compose config` passes on all four renamed files
- a tree-wide grep for `spark353`, `*_3.5.3`, `SPARK_VERSION=3.5.3` and
`docker.spark.version>3.5.3` returns nothing
- `bot.yml` parses, the three shell scripts pass `bash -n`, both poms pass
`xmllint`
- `spark-3.5.9-bin-hadoop3.tgz` is served by `archive.apache.org`, which
`spark_base/Dockerfile` fetches from, and by `dlcdn.apache.org`
- `build_docker_images.sh` picks the Java base and the hadoop-aws pairing by
major.minor, so a patch bump crosses no branch in the script
</details>
### Impact
Docker demo and the `hudi-integ-test` docker-cluster ITs only. No production
code paths change. All nine ITs on that stack are `@Disabled`, so nothing that
currently exercises the containers changes behaviour.
### Risk Level
**low** once the images exist. Until then the `integration-tests` check is
red by design.
### Documentation Update
`docker/README.md` is updated here. The website docker demo page lives on
the asf-site branch; if it names the compose files it needs a separate PR after
this lands.
### Contributor's checklist
- [x] Read through [contributor's
guide](https://hudi.apache.org/contribute/how-to-contribute)
- [x] Enough context is provided in the sections above
- [x] Adequate tests were added if applicable
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
