shangeyao commented on PR #11174: URL: https://github.com/apache/seatunnel/pull/11174#issuecomment-4862038943
Hi @DanielLeens, thanks again for the detailed review. ## CI update (head `7fb1071ad`) The dedicated Spark 4.1 merge gates are now green on fork run `28495262009`: | Job | Status | |-----|--------| | `spark-4-1-unit-test` | ✅ | | `spark-4-1-it` | ✅ | | `dist-spark41` | ✅ | The top-level `Build` is still red because of unrelated lanes (e.g. `Dead links`, some connector IT timeouts), but the Spark 4.1 path you previously asked to be validated is passing end to end. ## What changed since `80fec7880` The remaining `WrappedArray$ofRef` failure in `spark-4-1-it` was caused by `seatunnel-translation-spark-common` still being compiled against Spark 3.3 / Scala 2.12 and shaded into the 4.1 starter jar. The latest commit: - moves the common translation runtime (`execution/`, `serialization/`, `utils/`) into `seatunnel-translation-spark-4.1` and rebuilds it against Spark 4.1 / Scala 2.13 - adapts row converters away from Scala 2.12-only types (`WrappedArray`, `HashTrieMap`) - excludes the entire `seatunnel-translation-spark-common` artifact from the shaded 4.1 starter jar - fixes `dist-spark41` by using `-Dmaven.test.skip=true` (spark41 tarball intentionally does not ship transforms-v2) ## On the transform support boundary (Issue 1) You are right that the current contract is inconsistent. This is intentional for the first merge slice, but the docs have not caught up yet: - **Runtime / packaging**: `seatunnel-transforms-v2` is **not** bundled in `-spark41-bin` because transform plugins are still built against Scala 2.12 and conflict with Spark 4.1's Scala 2.13 runtime. - **E2E**: `FakeSpark41IT` exercises a minimal **source → sink** path (`fake_to_assert_spark41.conf`) without a transform stage, matching what the spark41 tarball actually ships today. - **Docs**: `docs/en/engines/spark.md` (and zh) still incorrectly list `seatunnel-transforms-v2` as bundled — I agree this needs to be corrected. **Proposed next step (Option B from your review):** update en/zh docs and the quick-start example to clearly state that Spark 4.1 currently supports the minimal source/sink path, and that transform support is tracked separately (design #11184, Phase 2). I can push that doc-alignment commit in this PR if you prefer, or as an immediate follow-up. Full transform E2E on Spark 4.1 (Option A) depends on making `seatunnel-transforms-v2` Scala 2.13 compatible, which is a larger follow-up. ## Upstream sync This branch is currently **16 commits ahead / 41 commits behind** `apache/seatunnel:dev`. I plan to rebase onto the latest `dev` next so CI signal is not mixed with upstream drift. Please let me know whether you would like the doc/contract alignment included in this PR before merge, or if the current CI-green Spark 4.1 source/sink path is sufficient for a first merge with docs fixed immediately after. Thanks! -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
