deepakpanda93 commented on code in PR #19489:
URL: https://github.com/apache/hudi/pull/19489#discussion_r3713200799
##########
docker/README.md:
##########
@@ -63,32 +64,39 @@ To build the Docker demo images with `docker` directly,
rather than through the
`build_local_docker_images.sh` above, run the script from under
`<HUDI_REPO_DIR>/docker`:
```shell
-# With no flags, builds Hadoop 2.8.4 / Spark 3.5.3 / Hive 2.3.10, matching
-# docker-compose_hadoop284_hive2310_spark353_{amd64,arm64}.yml
-./build_docker_images.sh
+# Builds Hadoop 3.3.4 / Spark 3.5.3 / Hive 3.1.3, matching
+# docker-compose_hadoop334_hive313_spark353_{amd64,arm64}.yml, which is the
set setup_demo.sh runs
+./build_and_publish_docker_images.sh
```
-You can override the Hadoop, Spark, and Hive versions from the command line.
If you plan to use `setup_demo.sh`,
-build the image set matching the default compose files first. For other flows,
use one of the supported version
-combinations under `docker/compose`.
+To build one of the other version combinations under `docker/compose`, edit
the version block near the top of the
Review Comment:
You are right, and the flags are the better instruction — fixed.
The script does take `--hadoop-version`, `--spark-version` and
`--hive-version` (option parsing at lines 84-86), so telling readers to hand
edit the source was both inconsistent with the example below and worse advice.
That section now reads:
```shell
./build_and_publish_docker_images.sh --hadoop-version 3.3.4 --spark-version
3.5.3 --hive-version 3.1.3
```
with a note that those are the defaults and each can be set on its own. No
`edit the version block` instruction remains anywhere in the README.
##########
docker/README.md:
##########
@@ -63,32 +64,39 @@ To build the Docker demo images with `docker` directly,
rather than through the
`build_local_docker_images.sh` above, run the script from under
`<HUDI_REPO_DIR>/docker`:
```shell
-# With no flags, builds Hadoop 2.8.4 / Spark 3.5.3 / Hive 2.3.10, matching
-# docker-compose_hadoop284_hive2310_spark353_{amd64,arm64}.yml
-./build_docker_images.sh
+# Builds Hadoop 3.3.4 / Spark 3.5.3 / Hive 3.1.3, matching
+# docker-compose_hadoop334_hive313_spark353_{amd64,arm64}.yml, which is the
set setup_demo.sh runs
+./build_and_publish_docker_images.sh
```
-You can override the Hadoop, Spark, and Hive versions from the command line.
If you plan to use `setup_demo.sh`,
-build the image set matching the default compose files first. For other flows,
use one of the supported version
-combinations under `docker/compose`.
+To build one of the other version combinations under `docker/compose`, edit
the version block near the top of the
+script before running it:
```shell
-# Matches setup_demo.sh and
-# docker-compose_hadoop334_hive313_spark353_{amd64,arm64}.yml
-./build_docker_images.sh --hadoop-version 3.3.4 --spark-version 3.5.3
--hive-version 3.1.3
-
-# Another supported combination is
-# docker-compose_hadoop340_hive313_spark401_{amd64,arm64}.yml
-./build_docker_images.sh --hadoop-version 3.4.0 --spark-version 4.0.1
--hive-version 3.1.3
+HADOOP_VERSION="3.3.4"
+SPARK_VERSION="3.5.3"
+HIVE_VERSION="3.1.3"
```
+The base image follows from `SPARK_VERSION`, so setting it to a 4.x release
also switches the set to the Java 17
+base. If you plan to use `setup_demo.sh`, build the image set matching its
compose file first.
+
`setup_demo.sh` currently defaults to
`docker-compose_hadoop334_hive313_spark353_{amd64,arm64}.yml`. If you build a
different image set for the demo flow, update `COMPOSE_FILE_NAME` in
`setup_demo.sh` to point to the matching compose
file before running the script. Run `./setup_demo.sh dev` to use your locally
built images; a plain run pulls the
Docker Hub images over them.
-By default, the script builds images for the current machine architecture and
derives the version tag from the root
-`pom.xml`. Use `--version-tag` to set an explicit tag if needed.
+The script builds images for the current machine architecture and tags each
one `:latest` plus the Hudi version
+taken from the root `pom.xml`. Export `VERSION_TAG` to use a different second
tag:
Review Comment:
Both work, and the docs now say so along with the order they resolve in. The
script does:
```sh
if [ -n "$VERSION_TAG_ARG" ]; then # --version-tag
VERSION_TAG="$VERSION_TAG_ARG"
elif [ -z "${VERSION_TAG:-}" ]; then # exported VERSION_TAG
VERSION_TAG=$(get_hudi_project_version ...)
fi
```
so it is flag, then environment variable, then the version from the root
`pom.xml`. Confirmed by running it rather than only reading it:
| invocation | resulting tag |
| --- | --- |
| `--version-tag from-flag` | `from-flag` |
| `VERSION_TAG=from-env` | `from-env` |
| both together | `from-flag` |
The README now documents both forms side by side and states that the flag
wins.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
