slfan1989 opened a new pull request, #2408:
URL: https://github.com/apache/auron/pull/2408
### Which issue does this PR close?
Closes #2407
### Rationale for this change
The Docker build example generated by `auron-build.sh --help` used Spark 4.1
together with the `centos7` image.
The `centos7` image only provides JDK 8, while Spark 4.x requires JDK 17 or
later. The example also enabled Iceberg and Hudi for Spark versions with which
those integrations are not compatible.
As a result, users following the documented example encountered Java version
or integration compatibility errors before completing the build.
### What changes are included in this PR?
- Replace the single Docker build example with explicit examples for
Spark 3.5, Spark 4.0, and Spark 4.1.
- Use the `rockylinux8` image for these examples because it supports the
JDK selection required by Spark 4.x.
- Use Scala 2.12 for Spark 3.5 and Scala 2.13 for Spark 4.x.
- Include only compatible third-party integrations in each example:
- Spark 3.5 includes Iceberg 1.10.1 and Hudi 0.15.
- Spark 4.0 includes Iceberg 1.10.1 but excludes Hudi.
- Spark 4.1 excludes both Iceberg and Hudi.
- Add a build compatibility matrix to `CONTRIBUTING.md`.
- Document the Spark, Scala, JDK, Docker image, Flink, Celeborn,
Uniffle, Paimon, Iceberg, and Hudi combinations.
- Update the existing Docker build example in `CONTRIBUTING.md` to use
`rockylinux8`.
- Document that the compatibility matrix and Shell help examples should
be updated together.
### Are there any user-facing changes?
Yes.
The help output now provides separate Docker build examples for Spark
3.5, Spark 4.0, and Spark 4.1. Each example uses an appropriate Scala,
JDK, Docker image, and third-party integration combination.
Contributors can also consult the compatibility matrix in
`CONTRIBUTING.md` before selecting build arguments.
### How was this patch tested?
> ./auron-build.sh --help
```
Usage: ./auron-build.sh [OPTIONS] <maven build options>
Build Auron project with specified Maven profiles
Options:
--pre Activate pre-release profile
--release Activate release profile
--clean <true|false> Clean before build (default: true)
--skiptests <true|false> Skip unit tests (default: true)
--sparktests <true|false> Run spark tests (default: false)
--docker <true|false> Build in Docker environment (default: false)
--threads <N|NC> Maven build threads (e.g. 1, 4, 1C). Default:
local unset, docker 8
--image <NAME> Docker image to use (e.g.
centos7,ubuntu24,rockylinux8,debian11,azurelinux3, default: centos7)
--sparkver <VERSION> Specify Spark version (e.g.
3.0,3.1,3.2,3.3,3.4,3.5,4.0,4.1)
--flinkver <VERSION> Specify Flink version (e.g. 1.18)
--scalaver <VERSION> Specify Scala version (e.g. 2.12,2.13)
--celeborn <VERSION> Specify Celeborn version (e.g. 0.5,0.6)
--uniffle <VERSION> Specify Uniffle version (e.g. 0.10)
--paimon <VERSION> Specify Paimon version (e.g. 1.2)
--iceberg <VERSION> Specify Iceberg version (e.g. 1.10.1)
--hudi <VERSION> Specify Hudi version (e.g. 0.15)
-h, --help Show this help message
Examples:
./auron-build.sh --pre --sparkver 4.1 --scalaver 2.13 -DskipBuildNative
# Spark 3.5 with all supported third-party integrations
./auron-build.sh --docker true --image rockylinux8 --clean true
--skiptests true --release --sparkver 3.5 --scalaver 2.12 --flinkver 1.18
--celeborn 0.6 --uniffle 0.10 --paimon 1.2 --iceberg 1.10.1 --hudi 0.15
# Spark 4.0 (Hudi is not supported)
./auron-build.sh --docker true --image rockylinux8 --clean true
--skiptests true --release --sparkver 4.0 --scalaver 2.13 --flinkver 1.18
--celeborn 0.6 --uniffle 0.10 --paimon 1.2 --iceberg 1.10.1
# Spark 4.1 (Iceberg and Hudi are not supported)
./auron-build.sh --docker true --image rockylinux8 --clean true
--skiptests true --release --sparkver 4.1 --scalaver 2.13 --flinkver 1.18
--celeborn 0.6 --uniffle 0.10 --paimon 1.2
```
### Was this patch authored or co-authored using generative AI tooling?
- [ ] Yes
- [ ] No
If yes, include: `Generated-by: <tool name and version>`
ASF guidance: https://www.apache.org/legal/generative-tooling.html
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
