This is an automated email from the ASF dual-hosted git repository.
xuba pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/amoro.git
The following commit(s) were added to refs/heads/master by this push:
new 99ecf533c [AMORO-3972][Core] Upgrade default Spark version from 3.3 to
3.5 (#3975)
99ecf533c is described below
commit 99ecf533c8613dd8297a60df1ab003bd07c5d23d
Author: zhan7236 <[email protected]>
AuthorDate: Thu Nov 27 14:22:12 2025 +0800
[AMORO-3972][Core] Upgrade default Spark version from 3.3 to 3.5 (#3975)
* [AMORO-3972][Core] Upgrade default Spark version from 3.3 to 3.5
This PR upgrades the default Spark version from 3.3 to 3.5 in the Amoro
project.
Changes:
- pom.xml: Update spark.version from 3.3.4 to 3.5.7 and spark.major.version
from 3.3 to 3.5
- pom.xml (hadoop2 profile): Update spark.version from 3.3.4 to 3.5.7 and
spark.major.version from 3.3 to 3.5
- docker/build.sh: Update SPARK_VERSION from 3.3.3 to 3.5.7
- docker/optimizer-spark/Dockerfile: Update ARG SPARK_VERSION from 3.3.3 to
3.5.7
- .github/workflows/docker-images.yml: Update Spark optimizer matrix from
3.3.3 to 3.5.7
- README.md: Update Spark optimizer default version documentation
Closes #3972
* Keep Spark 3.3 as default for hadoop2 profile
Address review feedback from @turboFei:
For hadoop2 profile, keep using the legacy Spark 3.3 by default
for better compatibility with Hadoop 2.x.
* Fix Spark version parameter name in README.md
Address review feedback from @turboFei:
Change -Dspark-optimizer.spark-version to -Dspark.version
This was missed in PR #3874.
---
.github/workflows/docker-images.yml | 12 ++++++------
README.md | 2 +-
docker/build.sh | 4 ++--
docker/optimizer-spark/Dockerfile | 2 +-
pom.xml | 4 ++--
5 files changed, 12 insertions(+), 12 deletions(-)
diff --git a/.github/workflows/docker-images.yml
b/.github/workflows/docker-images.yml
index 5b75fec68..9cd0bae55 100644
--- a/.github/workflows/docker-images.yml
+++ b/.github/workflows/docker-images.yml
@@ -186,7 +186,7 @@ jobs:
if: ${{ startsWith(github.repository, 'apache/') }}
strategy:
matrix:
- spark: [ "3.3.3" ]
+ spark: [ "3.5.7" ]
scala: [ "2.12.15" ]
steps:
- uses: actions/checkout@v3
@@ -212,11 +212,11 @@ jobs:
images: |
name=apache/amoro-spark-optimizer
tags: |
- type=ref,event=branch,enable=${{ matrix.spark == '3.3.3'
}},suffix=-snapshot
- type=ref,event=branch,enable=${{ matrix.spark == '3.3.3'
}},suffix=-snapshot-spark3.3
- type=raw,enable=${{ matrix.hadoop == '3.3.3' &&
startsWith(github.ref, 'refs/tags/v') }},value=latest
- type=semver,enable=${{ matrix.spark == '3.3.3'
}},pattern={{version}}
- type=semver,enable=${{ matrix.spark == '3.3.3'
}},pattern={{version}}, suffix=-spark3.3
+ type=ref,event=branch,enable=${{ matrix.spark == '3.5.7'
}},suffix=-snapshot
+ type=ref,event=branch,enable=${{ matrix.spark == '3.5.7'
}},suffix=-snapshot-spark3.5
+ type=raw,enable=${{ matrix.hadoop == '3.5.7' &&
startsWith(github.ref, 'refs/tags/v') }},value=latest
+ type=semver,enable=${{ matrix.spark == '3.5.7'
}},pattern={{version}}
+ type=semver,enable=${{ matrix.spark == '3.5.7'
}},pattern={{version}}, suffix=-spark3.5
- name: Print tags
run: echo '${{ steps.meta.outputs.tags }}'
diff --git a/README.md b/README.md
index 9ae32f313..7ec17f5f4 100644
--- a/README.md
+++ b/README.md
@@ -126,7 +126,7 @@ Amoro is built using Maven with JDK 8, 11 and 17(required
for `amoro-format-mixe
* Build with hadoop 2.x(the default is 3.x) dependencies: `./mvnw clean
package -DskipTests -Phadoop2`
* Specify Flink version for Flink optimizer(the default is 1.20.0): `./mvnw
clean package -DskipTests -Dflink-optimizer.flink-version=1.20.0`
* If the version of Flink is below 1.15.0, you also need to add the
`-Pflink-optimizer-pre-1.15` parameter: `./mvnw clean package -DskipTests
-Pflink-optimizer-pre-1.15 -Dflink-optimizer.flink-version=1.14.6`
-* Specify Spark version for Spark optimizer(the default is 3.3.3): `./mvnw
clean package -DskipTests -Dspark-optimizer.spark-version=3.3.3`
+* Specify Spark version for Spark optimizer(the default is 3.5.7): `./mvnw
clean package -DskipTests -Dspark.version=3.5.7`
* Build `amoro-mixed-trino` module under JDK 17: `./mvnw clean package
-DskipTests -Pformat-mixed-format-trino,build-mixed-format-trino -pl
'amoro-format-mixed/amoro-mixed-trino' -am`.
* Build all modules: `./mvnw clean package -DskipTests
-Ptoolchain,build-mixed-format-trino`, besides you need config `toolchains.xml`
in `${user.home}/.m2/` dir with content below.
* Build a distribution package with all formats integrated: `./mvnw clean
package -Psupport-all-formats`
diff --git a/docker/build.sh b/docker/build.sh
index 213a6fc85..3a02e10d5 100755
--- a/docker/build.sh
+++ b/docker/build.sh
@@ -27,7 +27,7 @@ cd $CURRENT_DIR
AMORO_VERSION=`cat $PROJECT_HOME/pom.xml | grep 'amoro-parent' -C 3 | grep -Eo
'<version>.*</version>' | awk -F'[><]' '{print $3}'`
FLINK_VERSION=1.20.0
-SPARK_VERSION=3.3.3
+SPARK_VERSION=3.5.7
DEBIAN_MIRROR=http://deb.debian.org
APACHE_ARCHIVE=https://archive.apache.org/dist
FLINK_OPTIMIZER_JOB_PATH=amoro-optimizer/amoro-optimizer-flink/target/amoro-optimizer-flink-${AMORO_VERSION}-jar-with-dependencies.jar
@@ -50,7 +50,7 @@ Images:
Options:
--flink-version Flink binary release version, default is 1.20.0,
format must be x.y.z
- --spark-version Spark binary release version, default is 3.3.3,
format must be x.y.z
+ --spark-version Spark binary release version, default is 3.5.7,
format must be x.y.z
--apache-archive Apache Archive url, default is
https://archive.apache.org/dist
--debian-mirror Mirror url of debian, default is
http://deb.debian.org
--maven-mirror Mirror url of maven, default is
https://repo.maven.apache.org/maven2
diff --git a/docker/optimizer-spark/Dockerfile
b/docker/optimizer-spark/Dockerfile
index 9c1e17fc2..4c06576dd 100644
--- a/docker/optimizer-spark/Dockerfile
+++ b/docker/optimizer-spark/Dockerfile
@@ -14,7 +14,7 @@
# See the License for the specific language governing permissions and
# limitations under the License.
-ARG SPARK_VERSION=3.3.3
+ARG SPARK_VERSION=3.5.7
FROM apache/spark:${SPARK_VERSION}
diff --git a/pom.xml b/pom.xml
index b8ebbd120..8422fcf2b 100644
--- a/pom.xml
+++ b/pom.xml
@@ -126,8 +126,8 @@
<orc-core.version>1.9.7</orc-core.version>
<awssdk.version>2.24.12</awssdk.version>
<aliyun-sdk-oss.version>3.10.2</aliyun-sdk-oss.version>
- <spark.version>3.3.4</spark.version>
- <spark.major.version>3.3</spark.major.version>
+ <spark.version>3.5.7</spark.version>
+ <spark.major.version>3.5</spark.major.version>
<dropwizard.metrics.version>4.2.19</dropwizard.metrics.version>
<caffeine.version>2.9.3</caffeine.version>
<snakeyaml.version>2.2</snakeyaml.version>
