deepakpanda93 opened a new pull request, #19489:
URL: https://github.com/apache/hudi/pull/19489
### Describe the issue this Pull Request addresses
Supersedes #14101, which was opened against a
`docker/build_docker_images.sh` that has since been
rewritten on master. Rebasing it would have reverted that work, so this is a
fresh port of the same
idea onto the current script. All review comments from #14101 are addressed
here.
Publishing the Hudi demo images to the `apachehudi` Docker Hub organisation
is a manual
`docker push` per image today, as documented under *Upload Updated Image to
Repository on Docker
Hub*. There is no scripted path, so a release publishes nine images by hand
across two tags each.
### Summary and Changelog
`docker/build_and_publish_docker_images.sh` replaces
`docker/build_docker_images.sh` and does both
jobs, which is what @xushiyan asked for on #14101 rather than a second
script duplicating the image
list and version variables.
- `--publish <true|false>`, default `false`, so a plain run only ever writes
to the local Docker
daemon. Named rather than positional, so the command states what it is
about to do.
- `--hadoop-version`, `--spark-version`, `--hive-version`, `--version-tag`,
all optional.
- `--multi-arch <true|false>`, default `false`.
- The base image follows the Spark version, so a 4.x release selects
`base_java17`.
- Every build passes `HADOOP_VERSION`, `SPARK_VERSION`, `HIVE_VERSION` and
`BASE_IMAGE_TAG`. The
Dockerfiles carry differing `ARG` defaults, so without this an image can
get a name that does not
describe its contents.
- The push step reports a per-tag summary and exits non-zero if any push
failed, so a partial
publish is not read as success.
- `--help` prints the usage, including who publishing is for.
Two things worth reviewer attention:
**Default versions changed.** They were Hadoop 2.8.4 / Hive 2.3.10, but
`setup_demo.sh` runs
`docker-compose_hadoop334_hive313_spark353`, so a bare build produced a set
the demo never
referenced. Defaults are now Hadoop 3.3.4 / Hive 3.1.3 / Spark 3.5.3 to
match. The old combination
is still reachable with the version flags.
**Multi-arch without publishing.** `buildx` cannot leave a multi-platform
result nowhere: it either
pushes or loads. With `--publish true` it pushes, which is also what makes
each image visible to the
next one, since every image after the base starts `FROM` the previous one.
Without it the images are
loaded into the local **containerd** image store, using the builder from the
current docker context.
Both are required: the legacy image store cannot hold a multi-platform
image, and a
`docker-container` builder resolves `FROM` against registries only, so it
silently builds the later
images against whatever is already on Docker Hub rather than against the
run. The script checks for
the containerd store and stops with an explanation instead.
**One unrelated bug fixed because it blocked the above.** `base_java11`
copied
`/etc/hadoop/mapred-site.xml.template` unconditionally, a file Hadoop 3.x
does not ship, so Hadoop
3.x could not be built on the Java 11 base at all. The copy is now guarded.
The guard is POSIX
`[ -f ... ]` rather than the `[[ -f ... ]]` used by the legacy `base`
module, because the Dockerfile
`RUN` executes under `/bin/sh`, which is `dash` in these images; `[[` is not
a command there, so the
condition is always false and the copy would be skipped even when the file
exists. Happy to split
this into its own PR if reviewers prefer.
`docker/README.md` is updated throughout: a publishing section stating this
is a maintainer action,
the version and multi-arch flags, and what keeping multi-arch images locally
requires.
### Impact
Publishing the demo image set becomes one command for maintainers. A plain
build now produces the
images `setup_demo.sh` actually consumes. Hadoop 3.x is buildable on the
Java 11 base.
`build_docker_images.sh` is removed, so anyone invoking it by name will need
to switch. Nothing in
the repository referenced it other than the README.
### Risk Level
low
### Documentation Update
`docker/README.md` is updated in this PR. No Hudi website change needed.
### Verification
Argument handling and the resulting `docker`/`buildx` command lines were
checked by stubbing
`docker` on `PATH` and asserting what the script invokes:
| run | `docker build` | `buildx` | `buildx --push` | `buildx --load` |
`docker push` |
| --- | --- | --- | --- | --- | --- |
| *(default)* | 9 | 0 | 0 | 0 | 0 |
| `--publish true` | 9 | 0 | 0 | 0 | 18 |
| `--multi-arch true` | 0 | 9 | 0 | 9 | 0 |
| `--multi-arch true --publish true` | 0 | 9 | 9 | 0 | 0 |
| `--publish yes`, `--publish` with no value, `--multi-arch yes` | rejected,
exit 1 | | | | |
`--spark-version 4.0.1` selects `base_java17` and tags
`hudi-hadoop_3.4.0-base-java17`;
`--version-tag rc1` overrides the `pom.xml`-derived tag.
Real builds: `base_java11` and `datanode` were built for
`linux/amd64,linux/arm64` on Docker 28.1.1
with the containerd image store, confirming the Hadoop 3.3.4 guard and that
`datanode` resolves its
parent from the local store. The full nine-image multi-arch run was not
completed end to end, as it
is dominated by ~660 MB Apache mirror downloads per architecture.
### Contributor's checklist
- [x] Read through [contributor's
guide](https://hudi.apache.org/contribute/how-to-contribute)
- [x] Enough context is provided in the sections above
- [x] Adequate tests were added if applicable
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
