zhan7236 opened a new pull request, #3975:
URL: https://github.com/apache/amoro/pull/3975
## Why are the changes needed?
Currently, the default Spark version is 3.3. We should upgrade it to the
latest version that we support (Spark 3.5) to provide users with better
performance and features from newer Spark releases.
Close #3972.
## Brief change log
- `pom.xml`:
- Updated default `spark.version` from 3.3.4 to 3.5.7
- Updated default `spark.major.version` from 3.3 to 3.5
- Updated `hadoop2` profile to use Spark 3.5.7 instead of 3.3.4
- `build.sh`:
- Updated default `SPARK_VERSION` from 3.3.3 to 3.5.7
- Updated help text to reflect the new default version
- `docker/optimizer-spark/Dockerfile`:
- Updated ARG `SPARK_VERSION` from 3.3.3 to 3.5.7
- `docker-images.yml`:
- Updated Spark optimizer matrix version from 3.3.3 to 3.5.7
- Updated Docker image tags from `spark3.3` to `spark3.5`
- `README.md`:
- Updated documentation to reflect the new default Spark version for Spark
optimizer
## How was this patch tested?
- [ ] Add some test cases that check the changes thoroughly including
negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [x] Run test locally before making a pull request
Verified Maven properties are correctly set:
```bash
$ ./mvnw help:evaluate -Dexpression=spark.version -q -DforceStdout
3.5.7
$ ./mvnw help:evaluate -Dexpression=spark.major.version -q -DforceStdout
3.5
$ ./mvnw help:evaluate -Phadoop2 -Dexpression=spark.version -q -DforceStdout
3.5.7
```
The project already has existing Spark 3.5 support via the `spark-3.5` Maven
profile, so this change only updates the default version.
## Documentation
- Does this pull request introduce a new feature? (no)
- If yes, how is the feature documented? (not applicable)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
