shangeyao opened a new issue, #4409:
URL: https://github.com/apache/streampark/issues/4409
## Background
StreamPark currently targets **JDK 8** as the compile baseline and
recommended **Console runtime**:
- Root `pom.xml`: `<project.build.jdk>1.8</project.build.jdk>`
- Docker image (`docker/Dockerfile`): `JAVA_MAJOR_VERSION=8`,
`JAVA_HOME=/usr/lib/jvm/jdk8`
- Assembly scripts: classpath includes `${JAVA_HOME}/jre/lib` (removed in
JDK 9+ layout)
- JVM options: PermGen flags, legacy GC logging (`-XX:+PrintGCDetails`)
- SSO stack pinned to **Pac4j 4.x / buji-pac4j 5.x** with explicit "for jdk
8" comments
CI already partially validates JDK 11:
| Workflow | JDK 8 | JDK 11 |
|----------|-------|--------|
| `backend.yml` build | ✅ matrix | ✅ matrix |
| `unit-test.yml` | ✅ matrix | ✅ matrix |
| `e2e.yml` build image | ✅ | — |
| `e2e.yml` run tests | — | ✅ |
| `docker-push.yml` | ✅ (1.8) | — |
This means **compilation and unit tests pass on JDK 11 today**, but
**official runtime, Docker distribution, docs, and startup scripts still assume
JDK 8**.
### Why migrate to JDK 11?
1. **Upstream alignment**: Apache Flink 2.x and Spark 3.x/4.x increasingly
assume JDK 11+ for job runtimes; keeping Console on JDK 8 creates a growing gap.
2. **Security & support**: JDK 8 public updates ended for most vendors; JDK
11 is an LTS release with long support windows.
3. **Dependency unlock**: Pac4j, Spring Boot ecosystem, and modern libraries
move forward on JDK 11+.
4. **CI simplification**: Reduce dual-JDK maintenance once JDK 11 becomes
the single baseline.
5. **Preparation for future**: JDK 17/21 upgrades are easier from 11 than
from 8.
### Important distinction: Console JDK ≠ Job JDK
StreamPark Console orchestrates Flink/Spark jobs that may run on **different
JDK versions** via engine-side `JAVA_HOME` (`flink-env.sh`, `spark-env.sh`,
Yarn container env). Migrating Console to JDK 11 **does not require** all
Flink/Spark clusters to upgrade simultaneously.
---
## Goals
- [ ] Make **JDK 11 the minimum required runtime** for StreamPark Console
(standalone, Docker, K8s deployment).
- [ ] Update Maven compile baseline to **Java 11** (`source`/`target` or
`--release 11`).
- [ ] Fix JDK 11 startup/runtime issues (module access, classpath layout,
JVM flags).
- [ ] Update CI, Docker, docs, and release artifacts to JDK 11.
- [ ] Document Console JDK vs Flink/Spark job JDK requirements clearly.
## Non-Goals (initial phase)
- Migrating to JDK 17 or 21 (may follow as a separate proposal).
- Forcing all Flink/Spark **job** runtimes to JDK 11 (engine env controls
this).
- Upgrading Spring Boot major version (stay on 2.7.x unless required for JDK
11).
- Changing Scala toolchain (see #4408 — orthogonal but can be coordinated).
---
## Current State Inventory
### Build configuration
| Location | Current | Target |
|----------|---------|--------|
| `pom.xml` `project.build.jdk` | `1.8` | `11` |
| `scala-maven-plugin` `-target:jvm-*` | `1.8` | `11` |
| Spotless Eclipse formatter | `1.8` compliance | `11` |
| Module POM overrides (e.g. elasticsearch connectors, e2e) |
`maven.compiler.source=8` | `11` |
| `streampark-spark-cli/create_app.sh` template | `1.8` | `11` |
### Runtime / assembly scripts
| File | JDK 8 assumption | Action |
|------|------------------|--------|
| `streampark.sh` | `${JAVA_HOME}/jre/lib` on classpath | Use unified
`$JAVA_HOME/lib` or drop JDK lib from classpath (app uses fat lib dir) |
| `jvm_opts.sh` | PermGen (via pom filter), `-XX:+PrintGC*` | Remove
PermGen; use `-Xlog:gc*` (JDK 11 unified logging) |
| `jvm_opts.sh` | `--add-opens` commented out | **Enable by default** on JDK
11+ |
| `setclasspath.sh` | JRE_HOME fallback | Validate JDK 11 layout |
| `docker/Dockerfile` | `JAVA_MAJOR_VERSION=8` | Default to `11`;
multi-stage optional |
| `streampark-env.sh` | Generic JAVA_HOME comment | Document JDK 11
requirement |
### Code requiring JDK 11 attention
| Area | Issue | Mitigation |
|------|-------|------------|
| `ClassLoaderUtils.addURL` | Reflective access to `jdk.internal.loader` /
`ucp` | Enable `--add-opens java.base/jdk.internal.loader=ALL-UNNAMED`;
long-term: replace with `URLClassLoader` or `MethodHandles` (see #4399) |
| `ParameterCli` / job submission | Already appends `--add-opens` for
submitted jobs | Keep; ensure Console JVM also has opens |
| `javax.annotation.*` (`@PostConstruct`, `@Nonnull`) | Removed from JDK 11
module path | Add `jakarta.annotation-api` or `javax.annotation-api` dependency
explicitly |
| `javax.mail` (1.4.7) | Legacy Java EE | Verify JDK 11 compatibility;
consider `jakarta.mail` in future |
| `sun.security.krb5.debug` system properties | Still valid | No change |
| Hadoop/YARN client libs | Jersey / JAX-RS classpath issues on JDK 11 |
Integration test with Hadoop 3.3.x; document `--add-opens` if needed |
### Dependencies pinned for JDK 8
| Dependency | Current | JDK 11 opportunity |
|------------|---------|-------------------|
| Pac4j | 4.5.7 | Evaluate 5.x (requires testing SSO flows: CAS, OIDC, SAML)
|
| buji-pac4j | 5.0.1 | Align with Pac4j upgrade |
| Spring Boot | 2.7.11 | Already supports JDK 11 — no major bump required |
---
## Proposed Target Architecture
### 1. Two-tier JDK model (document clearly)
```
┌─────────────────────────────────────┐
│ StreamPark Console (JDK 11+) │ ← this proposal
│ Spring Boot / Shiro / MyBatis │
└──────────────┬──────────────────────┘
│ submits / monitors
▼
┌─────────────────────────────────────┐
│ Flink / Spark Job Runtime │ ← controlled per environment
│ JAVA_HOME from flink-env.sh / │
│ spark-env.sh / K8s pod spec │
│ (JDK 8 / 11 / 17 depending on │
│ engine version) │
└─────────────────────────────────────┘
```
Publish a compatibility matrix in docs:
| Engine | Min job JDK | Recommended job JDK |
|--------|-------------|---------------------|
| Flink 1.12–1.17 | 8 | 8 or 11 |
| Flink 1.18–1.20 | 8 | 11 |
| Flink 2.x | 11 | 11 or 17 |
| Spark 2.x/3.x | 8 | 8 or 11 |
| Spark 4.x | 17 | 17 |
### 2. Default JVM options (Console)
Proposed `jvm_opts.sh` baseline for JDK 11:
```bash
-server
-Xms1g -Xmx1g
-XX:+ExitOnOutOfMemoryError
-XX:+HeapDumpOnOutOfMemoryError
-XX:+IgnoreUnrecognizedVMOptions
-Xlog:gc*:file=${APP_HOME}/logs/gc.log:time,uptime,level,tags:filecount=10,filesize=50M
# Required for dynamic classpath (ClassLoaderUtils, shims)
--add-opens java.base/jdk.internal.loader=ALL-UNNAMED
--add-opens jdk.zipfs/jdk.nio.zipfs=ALL-UNNAMED
```
Remove: `-XX:PermSize`, `-XX:MaxPermSize`, `-XX:+PrintGC*`.
### 3. Maven `--release 11`
Prefer `--release 11` over separate source/target to prevent accidental use
of JDK 11 APIs without awareness:
```xml
<project.build.jdk>11</project.build.jdk>
<!-- maven-compiler-plugin -->
<release>${project.build.jdk}</release>
```
### 4. Docker image
```dockerfile
ARG JAVA_MAJOR_VERSION=11
ENV JAVA_HOME=/usr/lib/jvm/jdk11
```
Provide migration note: users on JDK 8 Docker tag can pin to previous
release.
---
## Phased Execution Plan
### Phase 0 — Sign-off & compatibility policy (this issue)
- [ ] Committer agreement on JDK 11 as minimum Console runtime
- [ ] Define release note / upgrade guide for existing deployments
- [ ] Confirm Hadoop/YARN and K8s deployment smoke test environments
### Phase 1 — Runtime fixes (no compile bump yet)
- [ ] Enable `--add-opens` in `jvm_opts.sh` by default
- [ ] Fix `streampark.sh` classpath (remove invalid `jre/lib` path on JDK 11)
- [ ] Add explicit `javax.annotation-api` dependency to console-service
- [ ] Manual smoke test: start Console on JDK 11, login, list apps, submit
test job
**Deliverable**: Console runs on JDK 11 with bytecode still at 8
### Phase 2 — Compile baseline → 11
- [ ] Bump `project.build.jdk` to `11` in root POM
- [ ] Update Spotless formatter compliance to 11
- [ ] Update module-level compiler overrides (connectors, e2e)
- [ ] Fix any compilation errors from `--release 11`
**Deliverable**: `./mvnw clean install` on JDK 11 only
### Phase 3 — CI & Docker
- [ ] `docker/Dockerfile`: default JDK 11
- [ ] `docker-push.yml`: build with JDK 11
- [ ] `e2e.yml`: build image with JDK 11 (align build/run JDK)
- [ ] CI matrix: make JDK 11 primary; optionally keep JDK 8 as
`allow-failure` during transition, then remove
- [ ] Update `build.sh` / release scripts
### Phase 4 — Dependency refresh (optional, same release or follow-up)
- [ ] Evaluate Pac4j 5.x upgrade + SSO regression tests
- [ ] Review `javax.mail` / other Java EE artifacts
- [ ] Harden `ClassLoaderUtils` without relying on `ucp` reflection (#4399)
### Phase 5 — Documentation & release
- [ ] Update website docs (prerequisites, deployment, development guide)
- [ ] Update `README.md`, `streampark-env.sh` comments, Docker Hub
description
- [ ] Release notes: **breaking change** — JDK 11 required for Console
- [ ] Announce deprecation of JDK 8 Console runtime one release ahead (if
phased rollout desired)
---
## Risks & Mitigations
| Risk | Impact | Mitigation |
|------|--------|------------|
| Production users still on JDK 8 | High | Clear upgrade guide; one-release
deprecation window; Docker tag pinning |
| Hadoop/YARN client breakage on JDK 11 | Medium | E2E Yarn tests; document
required Hadoop version |
| SSO (Pac4j) regression | Medium | Dedicated SSO test plan before Pac4j
bump |
| `ClassLoaderUtils` reflection breaks on JDK 17+ | Medium | Fix ucp lookup
(#4399); `--add-opens` as interim |
| Shims classloader isolation regression | High | Submit/cancel/SQL verify
tests per Flink version |
| Third-party plugins/custom extensions compiled for 8 | Low | StreamPark
public API stays Java; bytecode 11 generally runs fine for consumers |
---
## Acceptance Criteria
1. `./mvnw -Pshaded,webapp,dist clean install` succeeds on **JDK 11 only**.
2. Console starts via `streampark.sh start` on JDK 11 without manual JVM
flags.
3. Official Docker image runs on JDK 11.
4. E2E suite passes with JDK 11 build + runtime.
5. Documentation states **JDK 11+ required** for Console.
6. Flink/Spark job JDK decoupling documented with compatibility matrix.
---
## Suggested Sub-Issues
1. `[Console] Fix startup scripts and JVM opts for JDK 11`
2. `[Console] Add javax.annotation-api and validate JDK 11 runtime smoke
test`
3. `[Build] Bump project.build.jdk to 11 and update Spotless/compiler config`
4. `[Docker] Migrate official image to JDK 11`
5. `[CI] Align workflows to JDK 11 baseline`
6. `[Docs] Document Console JDK 11 requirement and job JDK compatibility
matrix`
7. `[Console] Evaluate Pac4j upgrade for JDK 11 (optional)`
8. `[Common] Harden ClassLoaderUtils for JDK 11+ module system`
---
## Open Questions
1. **Deprecation window**: switch JDK 11 immediately in next major, or warn
one minor release first?
2. **CI matrix**: drop JDK 8 entirely, or keep as non-blocking canary for
one release?
3. **Bytecode for library modules**: `--release 11` everywhere, or keep
`streampark-flink-core` at 8 for user app compatibility?
4. **Pac4j upgrade**: bundle with JDK 11 migration or separate PR?
5. **Minimum Hadoop version** on JDK 11 Console: need explicit matrix?
6. **Coordination with #4408** (Scala removal): same release train or
independent?
---
## References
- Spring Boot 2.7 system requirements (Java 8–19): Console on 2.7.11 is JDK
11 ready
- Existing CI JDK 11 matrix in `backend.yml` / `unit-test.yml`
- `jvm_opts.sh` / `ParameterCli` `--add-opens` hints (already partially
implemented)
- Related: #4408 Remove Scala, #4399 ClassLoaderUtils JDK 9+ fix
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
