shangeyao opened a new issue, #11184: URL: https://github.com/apache/seatunnel/issues/11184
## Background The Spark engine layer in SeaTunnel has seen limited changes over the past few releases. Spark 4.1 support is being added in [#11174](https://github.com/apache/seatunnel/pull/11174), which introduces the core translation and starter modules. However, a broader review shows several release blockers, test gaps, and long-standing architectural debt that affect Spark 2.4, 3.3, and 4.1 alike. This issue tracks a proposed maintenance roadmap. Items are grouped by priority so we can split work into focused follow-up PRs. --- ## P0 — Spark 4.1 release readiness (recommended before merge of #11174) ### 1. Official binary distribution does not include Spark 4.1 `seatunnel-dist` assembly descriptors and dependencies only package Spark 2.4 and 3.3 starters: - `seatunnel-dist/src/main/assembly/assembly-bin.xml` - `seatunnel-dist/src/main/assembly/assembly-bin-ci.xml` - `seatunnel-dist/pom.xml` Users downloading the official release tarball cannot run Spark 4.1 jobs without building from source. **Suggested action:** - Add `start-seatunnel-spark-4.1-connector-v2.sh/.cmd` to assembly file sets - Add `seatunnel-spark-4.1-starter` jar to dependency sets - Ensure dist build activates the `jdk17` profile (or add a dedicated release job on JDK 17) ### 2. Documentation does not mention Spark 4.1 The following docs still only cover Spark 2.x and 3.x: - `docs/en|zh/getting-started/locally/quick-start-spark.md` - `docs/en|zh/engines/spark.md` - `docs/en|zh/engines/command/usage.mdx` - `docs/en|zh/getting-started/docker/docker.md` - `docs/en|zh/architecture/api-design/spark-translation-layer.md` **Suggested action:** - Document `start-seatunnel-spark-4.1-connector-v2.sh` - Document JDK 17 requirement for Spark 4.1 - Update architecture docs to describe the Spark 4.x translation module split ### 3. `EngineType` enum does not register Spark 4.1 `seatunnel-common/src/main/java/org/apache/seatunnel/common/constants/EngineType.java` only defines `SPARK2` and `SPARK3`. Tooling that relies on this enum (CLI helpers, packaging, etc.) cannot discover Spark 4.1. **Suggested action:** - Add `SPARK4` (or equivalent) with the correct starter jar and shell script names --- ## P1 — Test and CI coverage gaps | Gap | Current state | Suggested action | |-----|---------------|------------------| | Translation unit tests | Spark 3.3 has `SparkSinkTest` (type conversion + sink write) | Port equivalent tests to `seatunnel-translation-spark-4.1` | | Starter unit tests | Spark 3.3 runs on JDK 8; Spark 4.1 in-process tests fail on JDK 17 due to SLF4J/Log4j binding conflicts; CI only compiles | Fix test classpath logging, or rely on Docker IT for coverage | | E2E depth | Only `FakeSpark41IT` smoke test exists for Spark 4.1 | Add streaming and multi-table sink E2E (covers `MultiTableSinkTest` scenarios) | | Connector E2E matrix | Many ITs use `@DisabledOnContainer(SPARK_3_3)` or primarily run on `SPARK_2_4` | Gradually enable Spark 4.1 once stable; review connector-specific compatibility (e.g. Lance) | | E2E container strategy | Spark 2/3 use Bitnami images; Spark 4.1 uses official `apache/spark:4.1.2` with manual Master startup | Long-term: unify container strategy to reduce maintenance cost | | `SPARK_4_1` discovery | Only enabled when `RUN_SPARK_41_CONTAINER=true` | Confirm scheduled/full CI behavior matches product expectations | --- ## P2 — Architecture and long-standing technical debt ### 1. `seatunnel-spark-starter-common` is hard-wired to Spark 3.3 `seatunnel-spark-starter-common/pom.xml` depends on `seatunnel-translation-spark-3.3` and Spark 3.3 artifacts at compile time. The Spark 4.1 starter reuses this common module, relying on runtime substitution of version-specific translation classes with identical package names. This works today but is fragile. **Suggested action:** - Make `starter-common` depend only on `seatunnel-translation-spark-common` - Keep version-specific classes (`SparkSinkInjector`, etc.) in each starter/translation module ### 2. Significant code duplication across Spark translation modules Spark 3.3 and 4.1 translation modules contain ~27 nearly identical Java files. Spark 4.1 adds `useCommitCoordinator()` in `SeaTunnelBatchWrite`, which Spark 3.3 lacks — related to the long-open TODO in `MultiTableSinkTest` about closing the committer after Spark supports it. **Suggested action:** - Move shared logic into `seatunnel-translation-spark-common` - Use small version-specific adapters/subclasses for API differences - Consider backporting `useCommitCoordinator()` to 3.3 if the Spark 3.3 API supports it ### 3. Streaming still uses DStreams (`StreamingContext`) `SparkRuntimeEnvironment` creates a `StreamingContext` for streaming jobs. The translation layer already implements micro-batch via Spark SQL DataSource V2, but the starter path still depends on the legacy DStreams API, which is in maintenance mode in modern Spark. **Suggested action (long-term):** - Migrate streaming execution to Structured Streaming + micro-batch source - Remove `StreamingContext` dependency from starter-common ### 4. Open TODOs across Spark modules (unresolved for years) | Location | Issue | |----------|-------| | `SparkSinkInjector` (3.3 / 4.1) | Catalog table should be fetched in sink | | `SeaTunnelSourceTable` (3.3 / 4.1) | Waiting for all connectors to use `getProducedCatalogTables` | | `TypeConverterUtils` (common) | `// TODO: row kind` not handled | | `SparkRuntimeEnvironment` | Multi-jar submission still uses `--jars` workaround | | `SparkConfValidateCommand` | Config validation still uses legacy API | | `SparkStarter` | Job name override broken in Spark cluster deploy mode | | Spark 2.4 `SparkDataSourceWriter` | `// TODO close it` — potential resource leak on 2.4 path | ### 5. Scala version split - Spark 2.4 / 3.3: Scala 2.12 - Spark 4.1: Scala 2.13 Connectors with Spark/Scala dependencies need separate compatibility validation per Spark line. ### 6. Official Docker image is JDK 8 only `seatunnel-dist/src/main/docker/Dockerfile` uses `openjdk:8u342`. Spark 4.1 requires JDK 17, so Docker users cannot run Spark 4.1 even if the binary is packaged. **Suggested action:** - Provide a JDK 17-based image variant, or a multi-stage/multi-tag strategy --- ## P3 — Improvements that also benefit Spark 3.3 1. **No dedicated Spark 3.3 Docker smoke test** — Spark 4.1 adds `FakeSpark41IT`, but 3.3 still relies on generic `FakeIT` with `SPARK_2_4` as the primary Spark container 2. **Spark 3.3 starter unit tests fail on JDK 17** (module access issues) — hidden because CI main matrix uses JDK 8/11 3. **Netty version conflicts on Spark 3.3** — see [#9632](https://github.com/apache/seatunnel/pull/9632); shade strategy may need a systematic review 4. **`useCommitCoordinator` backport** — if supported on Spark 3.3+, backporting would improve aggregated committer lifecycle on 3.3 as well --- ## Proposed implementation roadmap ``` PR #11174 merge blockers └── dist packaging + docs + EngineType └── Port SparkSinkTest to 4.1 └── Extend E2E (streaming + multi-table) └── Decouple starter-common from Spark 3.3 └── Deduplicate translation 3.3/4.1 code └── StreamingContext → Structured Streaming └── JDK 17 Docker image + connector E2E matrix expansion ``` ### Minimal scope for #11174 follow-up - dist packaging - documentation (en + zh) - `EngineType` update - port `SparkSinkTest` to 4.1 - 1–2 additional E2E tests (streaming, multi-table) ### Larger Spark engine initiative (separate issue series) - Decouple `starter-common` from version-specific translation - Reduce translation module duplication - Structured Streaming migration - Docker multi-JDK support --- ## Related work - [#11174](https://github.com/apache/seatunnel/pull/11174) — Add Spark 4.1 support - Review discussion on missing Spark 4.1 E2E and CI coverage ## Labels (suggested) `improvement`, `translation`, `spark`, `tech-debt` --- Feedback welcome — happy to break this into smaller tracked sub-issues once we agree on priorities. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
