manoj-ragupathy opened a new pull request, #13021: URL: https://github.com/apache/gluten/pull/13021
## What changes are proposed in this pull request? > **Stacked on #13020.** This PR targets `main`, so its diff currently also shows the 6 commits from #13020. **Only the top 9 commits (from `Add Spark 4.2 shim module skeleton` onward) belong to this PR.** Once #13020 merges I will rebase and the diff will shrink to just those. Marked as draft until then. Second step of Spark 4.2.x support (#12569): add the `spark-4.2` build profile and the `shims/spark42` layer, mirroring what #11347 did for Spark 4.1. No unit-test module yet — that is the next PR — so this is buildable but not yet exercised by CI UT lanes. **Build plumbing** - New `spark-4.2` Maven profile (Spark `4.2.0`, Scala 2.13, Java 17, Delta disabled — see below), wired into the root `pom.xml`, `gluten-it`, and the package assembly. - `shims/spark42` module with its `SparkShimProvider` registered for Spark version `4.2`. **Shim implementations** — the Spark 4.2 side of the APIs abstracted in #13020, plus these: | Area | Spark 4.2 change | |---|---| | `BatchScanExecShim` | SPJ redesign: `KeyGroupedPartitioning` → `KeyedPartitioning`; `StoragePartitionJoinParams` and `KeyGroupedPartitionedScan` removed; `filteredPartitions` is now `Seq[Option[InputPartition]]`; runtime filters go through `PushDownUtils.pushRuntimeFilters` | | Columnar vector shims | signature changes in the `ColumnarArray` / `ColumnVector` hierarchy | | `MemoryStream` shim | constructor/signature change | | `PythonUDFRunner.writeUDFs` | argument list changed | | `postDriverMetrics` | now takes an explicit `Array[CustomTaskMetric]` | **CI** - Spark 4.2 entries in the change-detection matrix and the `install-spark-resources.sh` resource definitions, plus the Dockerfile lines that stage `/opt/shims/spark42/spark_home/`. - I deliberately grepped the new `install-spark-resources.sh` block for stray `spark41` references, since that was the bug fixed in #11973 during the 4.1 port. ### Two things worth flagging for reviewers **1. SPJ behaviour gap on 4.2 only.** `getCommonPartitionValues` returns `None` in the 4.2 shim. Spark 4.2 removed `spjParams` from the scan node and moved that logic into `EnsureRequirements` / `GroupPartitionsExec`, so there is no scan-level equivalent to read. The effect is that the *partially*-clustered-distribution refinement is disabled on 4.2; the base fully-clustered SPJ path is unaffected, and 3.4–4.1 behaviour is unchanged. Similarly `orderPartitions` no longer applies `joinKeyPositions` / `reducers` / replication — safe in practice because Gluten never populates those fields. I would appreciate a second opinion on whether this is an acceptable starting point or a blocker. **2. The 4.2 profile drops `-Pdelta`.** `delta-spark_4.2`, `iceberg-spark-runtime-4.2_2.13` and `hudi-spark4.2-bundle_2.13` do not exist on Maven Central yet. The lane can be enabled once they publish. Part of #12569. ## How was this patch tested? ```bash ./build/mvn -Pspark-4.2 -Pscala-2.13 -Pjava-17 -Pbackends-velox clean install -DskipTests ``` `BUILD SUCCESS`. #13020 (the base of this stack) is additionally compile-verified on Spark 3.4, 3.5, 4.0 and 4.1, confirming the new shim methods did not regress the existing versions. No unit tests run yet — `gluten-ut/spark42` arrives in the follow-up PR, and the 4.2 UT lanes additionally need the CI image rebuilt (see my note on #12569: `docker_image.yml` only rebuilds on a Sunday cron, so a committer needs to `workflow_dispatch` it before those lanes can pass). ## Was this patch authored or co-authored using generative AI tooling? Generated-by: GitHub Copilot CLI (Claude Opus 5) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
