This is an automated email from the ASF dual-hosted git repository. wenchen pushed a commit to branch update-release-infra-branch-3.5 in repository https://gitbox.apache.org/repos/asf/spark.git
commit 0737031fc190abf06443352f51f2c074c5e76de8 Author: Wenchen Fan <[email protected]> AuthorDate: Wed Jan 21 17:45:55 2026 +0800 [SPARK-XXXXX] Auto-detect Java version in release scripts ### What changes were proposed in this pull request? Updated the `init_java` function in `release-util.sh` to auto-detect and set the appropriate Java version based on the Spark version being built: - Spark < 4.0: Uses Java 8 - Spark >= 4.0: Uses Java 17 ### Why are the changes needed? This allows a single Docker image (with multiple Java versions installed) to build all Spark versions. Combined with updates to the master branch's Dockerfile to install both Java 8 and 17, this eliminates the need to maintain separate Dockerfiles for each branch. ### Does this PR introduce _any_ user-facing change? No. ### How was this patch tested? Manual testing by building releases with the updated scripts. ### Was this patch authored or co-authored using generative AI tooling? Yes. --- dev/create-release/release-build.sh | 16 +++++++++++++ dev/create-release/release-util.sh | 48 ++++++++++++++++++++++++++++++++++++- 2 files changed, 63 insertions(+), 1 deletion(-) diff --git a/dev/create-release/release-build.sh b/dev/create-release/release-build.sh index d16b9822b5a4..84fda7e82a1b 100755 --- a/dev/create-release/release-build.sh +++ b/dev/create-release/release-build.sh @@ -95,6 +95,22 @@ BASE_DIR=$(pwd) init_java init_maven_sbt +# Switch Java version based on release version. +# Spark 4.0+ requires Java 17, earlier versions use Java 8. +if [[ "$RELEASE_VERSION" > "3.5.99" ]]; then + if [ -n "$JAVA17_HOME" ] && [ -d "$JAVA17_HOME" ]; then + export JAVA_HOME="$JAVA17_HOME" + export PATH="$JAVA_HOME/bin:$PATH" + echo "Switched to Java 17 for Spark $RELEASE_VERSION" + fi +else + if [ -n "$JAVA8_HOME" ] && [ -d "$JAVA8_HOME" ]; then + export JAVA_HOME="$JAVA8_HOME" + export PATH="$JAVA_HOME/bin:$PATH" + echo "Switched to Java 8 for Spark $RELEASE_VERSION" + fi +fi + if [[ "$1" == "finalize" ]]; then if [[ -z "$PYPI_API_TOKEN" ]]; then error 'The environment variable PYPI_API_TOKEN is not set. Exiting.' diff --git a/dev/create-release/release-util.sh b/dev/create-release/release-util.sh index acdecadeee33..7df5faa32108 100755 --- a/dev/create-release/release-util.sh +++ b/dev/create-release/release-util.sh @@ -229,13 +229,59 @@ function is_dry_run { [[ $DRY_RUN = 1 ]] } -# Initializes JAVA_VERSION to the version of the JVM in use. +# Initializes JAVA_HOME and JAVA_VERSION based on Spark version. +# For Spark 4.0+, use Java 17. For earlier versions, use Java 8. +# This allows a single Docker image (with multiple Java versions) to build all Spark versions. function init_java { + # If JAVA_HOME is already set externally (e.g., via -j flag), use it + if [ -n "$JAVA_HOME" ] && [ -d "$JAVA_HOME" ]; then + echo "Using provided JAVA_HOME: $JAVA_HOME" + else + # Auto-detect appropriate Java version based on Spark version + # Try to get Spark version from pom.xml if we're in a Spark directory + local spark_version="" + # Extract project version, skipping the parent POM version + # The project version comes after the </parent> closing tag + if [ -f "pom.xml" ]; then + spark_version=$(awk '/<\/parent>/,/<version>/' pom.xml | grep '<version>' | head -1 | sed -e 's/.*<version>//' -e 's/<\/version>.*//' | tr -d '[:space:]') + elif [ -f "spark/pom.xml" ]; then + spark_version=$(awk '/<\/parent>/,/<version>/' spark/pom.xml | grep '<version>' | head -1 | sed -e 's/.*<version>//' -e 's/<\/version>.*//' | tr -d '[:space:]') + fi + + # Determine Java version based on Spark version + # Spark 4.0+ requires Java 17, earlier versions use Java 8 + if [[ -n "$spark_version" ]] && [[ "$spark_version" < "4.0" ]]; then + echo "Detected Spark version $spark_version - using Java 8" + if [ -n "$JAVA8_HOME" ] && [ -d "$JAVA8_HOME" ]; then + export JAVA_HOME="$JAVA8_HOME" + elif [ -d "/usr/lib/jvm/java-8-openjdk-amd64" ]; then + export JAVA_HOME="/usr/lib/jvm/java-8-openjdk-amd64" + elif [ -d "/usr/lib/jvm/java-8-openjdk-arm64" ]; then + export JAVA_HOME="/usr/lib/jvm/java-8-openjdk-arm64" + else + error "Java 8 is required for Spark $spark_version but not found." + fi + else + echo "Detected Spark version $spark_version - using Java 17" + if [ -n "$JAVA17_HOME" ] && [ -d "$JAVA17_HOME" ]; then + export JAVA_HOME="$JAVA17_HOME" + elif [ -d "/usr/lib/jvm/java-17-openjdk-amd64" ]; then + export JAVA_HOME="/usr/lib/jvm/java-17-openjdk-amd64" + elif [ -d "/usr/lib/jvm/java-17-openjdk-arm64" ]; then + export JAVA_HOME="/usr/lib/jvm/java-17-openjdk-arm64" + else + error "Java 17 is required for Spark $spark_version but not found." + fi + fi + fi + if [ -z "$JAVA_HOME" ]; then error "JAVA_HOME is not set." fi + JAVA_VERSION=$("${JAVA_HOME}"/bin/javac -version 2>&1 | cut -d " " -f 2) export JAVA_VERSION + echo "Using Java version: $JAVA_VERSION (JAVA_HOME=$JAVA_HOME)" } # Initializes MVN_EXTRA_OPTS and SBT_OPTS depending on the JAVA_VERSION in use. Requires init_java. --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
