shangeyao opened a new issue, #4422:
URL: https://github.com/apache/streampark/issues/4422
## Background
StreamPark's Spark integration (`-Pspark` profile) currently targets **Spark
3.1.2**, with version detection via `spark-submit --version` and shims loaded
through `SparkShimsProxy`. Several gaps block StreamPark 3.0:
1. **Outdated baseline**: Spark 3.1.x is far behind current LTS lines; Spark
3.5+ is the practical minimum for new deployments.
2. **Spark 4.x**: Spark 4 requires **JDK 17+ on the cluster**
(`spark-env.sh`), while StreamPark Console runs on JDK 11 — version detection
and job JDK propagation must be decoupled.
3. **Remove Scala alignment**: Post–#4418, Spark shims jars no longer use
`_2.12` artifact suffix; `SparkShimsProxy` must match Java-only jar names.
4. **Version detection fragility**: Relying solely on `spark-submit
--version` fails when cluster JDK differs from the Console service JDK (common
with Spark 4).
This issue tracks the **design and execution plan** for Spark version
baseline changes in StreamPark 3.0.
**Implementation PR:** #4420 — `[Spark] Baseline Spark 3.5+ and add Spark
4.x support`
**Depends on:** #4418 (Remove Scala) → #4417 (JDK 11 baseline)
**Independent of:** #4419 (Flink version track)
**Parent roadmap:** #4410
---
## Goals
- [ ] Baseline Spark at **3.5+**; reject Spark 2.x and 3.0–3.4 in
`SparkVersion.checkVersion()`.
- [ ] Support **Spark 4.x** env registration and job submission.
- [ ] Resolve cluster-side `JAVA_HOME` from `spark-env.sh` (Spark 4 needs
JDK 17+ on cluster).
- [ ] Update `SparkShimsProxy`, `YarnClient`, compile-time `spark.version`,
and frontend hints.
- [ ] Document Console JDK vs Spark job JDK requirements.
## Non-Goals
- Supporting Spark 2.x / 3.0–3.4 in 3.0.
- Upgrading StreamPark Console to JDK 17 (Console stays JDK 11; Spark 4 jobs
use cluster JDK 17).
- Migrating Spark modules to Java — handled by #4408 / #4418.
- Splitting into multiple PRs — **one consolidated PR** (#4420).
---
## Current Baseline (2.x)
| Area | Today |
|------|-------|
| `spark.version` (compile) | 3.1.2 |
| Supported runtime | Implicitly 2.x / 3.x (loose validation) |
| Spark 4.x | Not supported |
| Version detection | `spark-submit --version` stdout parsing |
| Job JDK | Not explicitly resolved from `spark-env.sh` |
| Shims jar matching | `_2.12` suffix expected |
---
## 3.0 Target State
| Area | Target |
|------|--------|
| **Supported 3.x** | 3.5, 3.5.x |
| **Supported 4.x** | 4.0, 4.x |
| **Rejected** | Spark 2.x, 3.0–3.4 |
| Compile-time `spark.version` | 3.5.4 |
| Version detection | `spark-core_*.jar` / `RELEASE` file first; fallback to
`spark-submit --version` |
| Job JDK | `SparkEnvUtils` parses `JAVA_HOME` from `spark-env.sh`;
propagated to `SparkLauncher` on YARN |
| Shims jar matching | No `_2.12` suffix (aligned with Remove Scala) |
---
## Proposed Architecture
### Two-tier JDK model
```
┌─────────────────────────────────────┐
│ StreamPark Console (JDK 11+) │ ← #4409 / #4417
└──────────────┬──────────────────────┘
│ submits / monitors
▼
┌─────────────────────────────────────┐
│ Spark Job Runtime │
│ JAVA_HOME from spark-env.sh │
│ Spark 3.5+ → JDK 11+ recommended │
│ Spark 4.x → JDK 17+ required │
└─────────────────────────────────────┘
```
### Key code changes (#4420)
| Component | Change |
|-----------|--------|
| `SparkVersion` | Accept 3.5+ and 4.x; parse `spark-core_*.jar` and
`RELEASE` before `spark-submit` |
| `SparkEnvUtils` | New utility: read `JAVA_HOME` from `spark-env.sh` |
| `YarnClient` | Pass resolved cluster `JAVA_HOME` to `SparkLauncher` |
| `SparkShimsProxy` | Match shims jars without `_2.12` suffix |
| `pom.xml` | `spark.version` 3.1.2 → 3.5.4; add
`commons-collections4.version=4.4` |
| Frontend `spark/home.ts` | Update env registration hints for 3.5+ / 4.x |
### Version detection priority
```
1. Parse spark-core_*.jar filename in $SPARK_HOME/jars/
2. Read $SPARK_HOME/RELEASE
3. Fallback: spark-submit --version (legacy path)
```
This avoids registration failures when Spark 4 is installed but
`spark-submit` cannot run under the Console JVM.
---
## Execution Plan
Delivered as **one PR** (#4420) on branch `feature/spark-3.5-4-support`:
1. Update `SparkVersion`, add `SparkEnvUtils` with unit tests
2. Update `YarnClient`, `SparkShimsProxy`, root POM, frontend hints
3. CI green on JDK 11 + Java Spark modules (#4418 merged first)
**Can run in parallel with:** #4419 (Flink version track) after #4418 merges.
---
## Risks & Mitigations
| Risk | Impact | Mitigation |
|------|--------|------------|
| Users on Spark 3.1–3.4 | High | 3.0 release notes; upgrade guide to 3.5+ |
| Spark 4 cluster JDK 17 not configured | Medium | `SparkEnvUtils`
validation + clear Console error messages |
| YARN submission JDK mismatch | Medium | Propagate `JAVA_HOME` from
`spark-env.sh` to `SparkLauncher` |
| Spark 4 API changes vs 3.5 compile dep | Medium | Shims proxy isolates
version differences; integration test on 4.x |
---
## Acceptance Criteria
1. `SparkVersionTest` and `SparkEnvUtilsTest` pass on JDK 11.
2. `./mvnw clean compile -Pfast -pl streampark-spark -am` succeeds.
3. Console registers Spark 3.5 and 4.x homes; version detection correct
without running `spark-submit` under Console JVM.
4. Spark job submits on YARN with Spark 4.x when `spark-env.sh` sets JDK 17.
5. Supported version matrix documented in upgrade guide.
---
## Open Questions
1. Spark **3.5.x** exact compile pin — 3.5.0 vs latest 3.5 LTS (currently
**3.5.4** in #4420)?
2. K8s Spark submission: propagate `JAVA_HOME` via pod spec — in scope for
#4420 or follow-up?
3. Spark module optional profile: document that `-Pspark` remains optional
in 3.0?
---
## References
- Parent roadmap: #4410
- Remove Scala: #4408 / #4418
- JDK 11 baseline: #4409 / #4417
- Flink version track: #4419 (parallel, independent)
- Implementation PR: #4420
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
