shangeyao commented on issue #11184: URL: https://github.com/apache/seatunnel/issues/11184#issuecomment-4816026918
Thanks @nzw921rx for raising this — I think a unified Spark distribution is a reasonable direction to explore. Spark jobs in SeaTunnel are mostly batch / micro-batch, and unlike CDC connectors, the runtime upgrade path is mainly about JDK + Spark version alignment rather than database protocol compatibility. So consolidating Spark 2.4 / 3.3 / 4.1 into one Spark-focused release could simplify operations in the long run. That said, I would still treat it as a **post-merge follow-up design**, not a change to the current #11174 boundary, for a few practical reasons: 1. **JDK baseline** — Spark 4.1 requires JDK 17+, while today's standard `-bin` tarball is still expected to work on JDK 8/11 for Spark 2.4 / 3.3 users. Moving everyone to a single package immediately would either raise the JDK floor for existing users, or force a more complex multi-JDK packaging story. 2. **Release risk** — the dual-tarball approach keeps the current release path unchanged and gives Spark 4.1 users an explicit artifact with a clear JDK 17 expectation. 3. **Review scope** — #11174 should stay focused on making Spark 4.1 usable first; packaging consolidation is a separate compatibility decision. If maintainers agree this is worth pursuing, I'd be happy to help draft a follow-up design issue after #11174 lands, covering: - whether the unified Spark distribution standardizes on JDK 17 - whether Spark 2.4 remains in scope - how existing `-bin` users migrate - whether connectors are shared once across Spark versions or still split by engine packaging So: **+1 on the direction**, but I'd keep **dual tarball for Phase 1** and revisit **single Spark distribution** as the next packaging iteration. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
