shangeyao opened a new pull request, #4420:
URL: https://github.com/apache/streampark/pull/4420
## Summary
- **Baseline Spark at 3.5+** — `SparkVersion.checkVersion()` accepts Spark
3.5+ and 4.x; rejects Spark 2.x and 3.0–3.4.
- **Spark 4.x env registration** — parse version from `spark-core_*.jar` /
`RELEASE` before running `spark-submit`; avoids failures when cluster JDK
differs from StreamPark service JDK.
- **`SparkEnvUtils`** — resolve cluster-side `JAVA_HOME` from `spark-env.sh`
(Spark 4 requires JDK 17+ on the cluster; StreamPark service can stay on JDK
11).
- **`YarnClient`** — propagate resolved `JAVA_HOME` to `SparkLauncher` for
YARN submissions.
- **`SparkShimsProxy`** — match shims jars without `_2.12` artifact suffix
(aligned with Remove Scala).
- Bump compile-time Spark dependency **3.1.2 → 3.5.4**; add missing
`commons-collections4.version` property required by Spark 3.5.
**Depends on:** #4418 (Remove Scala on JDK 11 baseline), which depends on
#4417.
## Supported Spark versions after this PR
| 3.x | 4.x |
|-----|-----|
| 3.5, 3.5.x | 4.0, 4.x |
## Test plan
- [x] Unit tests:
```bash
export JAVA_HOME=$(/usr/libexec/java_home -v 11)
./mvnw test -pl streampark-common \
-Dtest=SparkVersionTest,SparkEnvUtilsTest -Drat.skip=true
```
- [x] Spark modules compile:
```bash
./mvnw clean compile -Pfast -pl streampark-spark -am -DskipTests
```
- [ ] Register Spark 3.5 and Spark 4.x homes in Console and verify version
detection
- [ ] Submit Spark job on YARN with Spark 4.x (JDK 17 in `spark-env.sh`)
---
**AI Disclosure**
- Model: Claude Opus (Composer)
- Platform/Tool: Cursor
- Human Oversight: partially reviewed
- Prompt Summary: Spark 3.5+ baseline and Spark 4.x support on JDK 11 +
Remove Scala base, as a dedicated PR separate from Flink version work
Made with [Cursor](https://cursor.com)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
