This is an automated email from the ASF dual-hosted git repository.
slfan1989 pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/auron.git
The following commit(s) were added to refs/heads/master by this push:
new 858dc6ae [AURON #2407] [BUILD] Fix the invalid Spark 4.1 Docker build
example. (#2408)
858dc6ae is described below
commit 858dc6aef2fb72bc0aa19ab092ea0b1eafc7f64f
Author: slfan1989 <[email protected]>
AuthorDate: Sun Jul 26 13:18:26 2026 +0800
[AURON #2407] [BUILD] Fix the invalid Spark 4.1 Docker build example.
(#2408)
### Which issue does this PR close?
Closes #2407
### Rationale for this change
The Docker build example generated by `auron-build.sh --help` used Spark
4.1 together with the `centos7` image.
The `centos7` image only provides JDK 8, while Spark 4.x requires JDK 17
or later. The example also enabled Iceberg and Hudi for Spark versions
with which those integrations are not compatible.
As a result, users following the documented example encountered Java
version or integration compatibility errors before completing the build.
### What changes are included in this PR?
- Replace the single Docker build example with explicit examples for
Spark 3.5, Spark 4.0, and Spark 4.1.
- Use the `rockylinux8` image for these examples because it supports the
JDK selection required by Spark 4.x.
- Use Scala 2.12 for Spark 3.5 and Scala 2.13 for Spark 4.x.
- Include only compatible third-party integrations in each example:
- Spark 3.5 includes Iceberg 1.10.1 and Hudi 0.15.
- Spark 4.0 includes Iceberg 1.10.1 but excludes Hudi.
- Spark 4.1 excludes both Iceberg and Hudi.
- Add a build compatibility matrix to `CONTRIBUTING.md`.
- Document the Spark, Scala, JDK, Docker image, Flink, Celeborn,
Uniffle, Paimon, Iceberg, and Hudi combinations.
- Update the existing Docker build example in `CONTRIBUTING.md` to use
`rockylinux8`.
- Document that the compatibility matrix and Shell help examples should
be updated together.
### Are there any user-facing changes?
Yes.
The help output now provides separate Docker build examples for Spark
3.5, Spark 4.0, and Spark 4.1. Each example uses an appropriate Scala,
JDK, Docker image, and third-party integration combination.
Contributors can also consult the compatibility matrix in
`CONTRIBUTING.md` before selecting build arguments.
### How was this patch tested?
> ./auron-build.sh --help
```
Usage: ./auron-build.sh [OPTIONS] <maven build options>
Build Auron project with specified Maven profiles
Options:
--pre Activate pre-release profile
--release Activate release profile
--clean <true|false> Clean before build (default: true)
--skiptests <true|false> Skip unit tests (default: true)
--sparktests <true|false> Run spark tests (default: false)
--docker <true|false> Build in Docker environment (default: false)
--threads <N|NC> Maven build threads (e.g. 1, 4, 1C). Default:
local unset, docker 8
--image <NAME> Docker image to use (e.g.
centos7,ubuntu24,rockylinux8,debian11,azurelinux3, default: centos7)
--sparkver <VERSION> Specify Spark version (e.g.
3.0,3.1,3.2,3.3,3.4,3.5,4.0,4.1)
--flinkver <VERSION> Specify Flink version (e.g. 1.18)
--scalaver <VERSION> Specify Scala version (e.g. 2.12,2.13)
--celeborn <VERSION> Specify Celeborn version (e.g. 0.5,0.6)
--uniffle <VERSION> Specify Uniffle version (e.g. 0.10)
--paimon <VERSION> Specify Paimon version (e.g. 1.2)
--iceberg <VERSION> Specify Iceberg version (e.g. 1.10.1)
--hudi <VERSION> Specify Hudi version (e.g. 0.15)
-h, --help Show this help message
Examples:
./auron-build.sh --pre --sparkver 4.1 --scalaver 2.13 -DskipBuildNative
# Spark 3.5 with all supported third-party integrations
./auron-build.sh --docker true --image rockylinux8 --clean true
--skiptests true --release --sparkver 3.5 --scalaver 2.12 --flinkver 1.18
--celeborn 0.6 --uniffle 0.10 --paimon 1.2 --iceberg 1.10.1 --hudi 0.15
# Spark 4.0 (Hudi is not supported)
./auron-build.sh --docker true --image rockylinux8 --clean true
--skiptests true --release --sparkver 4.0 --scalaver 2.13 --flinkver 1.18
--celeborn 0.6 --uniffle 0.10 --paimon 1.2 --iceberg 1.10.1
# Spark 4.1 (Iceberg and Hudi are not supported)
./auron-build.sh --docker true --image rockylinux8 --clean true
--skiptests true --release --sparkver 4.1 --scalaver 2.13 --flinkver 1.18
--celeborn 0.6 --uniffle 0.10 --paimon 1.2
```
### Was this patch authored or co-authored using generative AI tooling?
- [ ] Yes
- [x] No
If yes, include: `Generated-by: <tool name and version>`
ASF guidance: https://www.apache.org/legal/generative-tooling.html
Signed-off-by: slfan1989 <[email protected]>
---
CONTRIBUTING.md | 26 ++++++++++++++++++++++++--
auron-build.sh | 35 ++++++++++++++++++++++++++---------
2 files changed, 50 insertions(+), 11 deletions(-)
diff --git a/CONTRIBUTING.md b/CONTRIBUTING.md
index 6d8e4d84..c91422e2 100644
--- a/CONTRIBUTING.md
+++ b/CONTRIBUTING.md
@@ -114,7 +114,7 @@ Auron provides a unified build script `auron-build.sh` that
supports both local
```bash
# Build inside Docker container
-./auron-build.sh --docker true --image centos7 --release \
+./auron-build.sh --docker true --image rockylinux8 --release \
--sparkver 3.5 --scalaver 2.12
```
@@ -123,13 +123,35 @@ Auron provides a unified build script `auron-build.sh`
that supports both local
Run `./auron-build.sh --help` to see all available options, including:
- `--pre` or `--release`: Build profile
-- `--sparkver`: Spark version (3.0, 3.1, 3.2, 3.3, 3.4, 3.5)
+- `--sparkver`: Spark version (3.0, 3.1, 3.2, 3.3, 3.4, 3.5, 4.0, 4.1)
- `--scalaver`: Scala version (2.12, 2.13)
- `--celeborn`, `--uniffle`, `--paimon`, `--iceberg`: Optional integrations
- `--skiptests`: Skip unit tests (default: true)
- `--sparktests`: Run Spark integration tests
- `--threads`: Maven build threads (e.g. 1, 4, 1C). Defaults to
single-threaded local builds; Docker defaults to 8 unless overridden.
+### Build Compatibility Matrix
+
+The examples printed by `./auron-build.sh --help` use the following documented
build
+configurations based on the current compatibility checks. A dash (`—`) means
that the
+integration is not supported for that Spark version and must not be passed to
+`auron-build.sh`.
+
+| Spark | Scala | JDK | Docker image | Flink | Celeborn | Uniffle | Paimon |
Iceberg | Hudi |
+| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
+| 3.5 | 2.12 | 8+ | `rockylinux8` | 1.18 | 0.6 | 0.10 | 1.2 | 1.10.1 | 0.15 |
+| 4.0 | 2.13 | 17+ | `rockylinux8` | 1.18 | 0.6 | 0.10 | 1.2 | 1.10.1 | — |
+| 4.1 | 2.13 | 17+ | `rockylinux8` | 1.18 | 0.6 | 0.10 | 1.2 | — | — |
+
+Spark 4.x requires Scala 2.13 and JDK 17 or later. In Docker mode,
+`auron-build.sh` selects JDK 17 for Spark 4.x, and the `rockylinux8` image
applies that
+selection. The `centos7` image only provides JDK 8 and must not be used for
Spark 4.x.
+
+Iceberg 1.10.1 supports Spark 3.4, 3.5, and 4.0 in Auron. Spark 4.1 support
requires a
+future upgrade to Iceberg 1.11.0 or later. Hudi 0.15 supports Spark 3.0
through 3.5.
+When a supported version or compatibility rule changes, update this matrix and
the
+examples in `auron-build.sh` together.
+
### Running Tests
By default, the build script skips unit tests (`--skiptests true`). To run
them, you must explicitly set `--skiptests false`, as shown in the examples
below:
diff --git a/auron-build.sh b/auron-build.sh
index 2fc673a9..2fffcabf 100755
--- a/auron-build.sh
+++ b/auron-build.sh
@@ -72,16 +72,33 @@ print_help() {
echo "Examples:"
echo " $0 --pre --sparkver ${SUPPORTED_SPARK_VERSIONS[*]: -1}" \
"--scalaver ${SUPPORTED_SCALA_VERSIONS[*]: -1} -DskipBuildNative"
- echo " $0 --docker true --image ${SUPPORTED_OS_IMAGES[*]:0:1}" \
+ echo " # Spark 3.5 with all supported third-party integrations"
+ echo " $0 --docker true --image rockylinux8" \
"--clean true --skiptests true --release" \
- "--sparkver ${SUPPORTED_SPARK_VERSIONS[*]: -1}" \
- "--flinkver ${SUPPORTED_FLINK_VERSIONS[*]: -1}" \
- "--scalaver ${SUPPORTED_SCALA_VERSIONS[*]: -1}" \
- "--celeborn ${SUPPORTED_CELEBORN_VERSIONS[*]: -1}" \
- "--uniffle ${SUPPORTED_UNIFFLE_VERSIONS[*]: -1}" \
- "--paimon ${SUPPORTED_PAIMON_VERSIONS[*]: -1}" \
- "--iceberg ${SUPPORTED_ICEBERG_VERSIONS[*]: -1}" \
- "--hudi ${SUPPORTED_HUDI_VERSIONS[*]: -1}"
+ "--sparkver 3.5 --scalaver 2.12" \
+ "--flinkver 1.18" \
+ "--celeborn 0.6" \
+ "--uniffle 0.10" \
+ "--paimon 1.2" \
+ "--iceberg 1.10.1" \
+ "--hudi 0.15"
+ echo " # Spark 4.0 (Hudi is not supported)"
+ echo " $0 --docker true --image rockylinux8" \
+ "--clean true --skiptests true --release" \
+ "--sparkver 4.0 --scalaver 2.13" \
+ "--flinkver 1.18" \
+ "--celeborn 0.6" \
+ "--uniffle 0.10" \
+ "--paimon 1.2" \
+ "--iceberg 1.10.1"
+ echo " # Spark 4.1 (Iceberg and Hudi are not supported)"
+ echo " $0 --docker true --image rockylinux8" \
+ "--clean true --skiptests true --release" \
+ "--sparkver 4.1 --scalaver 2.13" \
+ "--flinkver 1.18" \
+ "--celeborn 0.6" \
+ "--uniffle 0.10" \
+ "--paimon 1.2"
exit 0
}
