akshaytayal opened a new pull request, #13126:
URL: https://github.com/apache/gluten/pull/13126
## What changes were proposed in this pull request?
This PR adds the **Spark 4.2 shim layer and build profile** to Gluten (Velox
backend), as part of the umbrella effort to support Apache Spark 4.2
(GLUTEN-12569).
It is built on top of the already-merged Spark 4.2 compatibility fixes and
introduces:
- **`shims/spark42` module skeleton** and its registration in
`shims/pom.xml`.
- **`spark-4.2` build profile** in the root `pom.xml` and in
`tools/gluten-it`.
- **Shim implementations** for Spark 4.2:
- New shim method implementations (`SparkShimProvider`, `Spark42Shims`).
- `MemoryStream` shim adaptation.
- `PythonUDFRunner.writeUDFs` call adaptation.
- Columnar vector shims.
- `BatchScanExec` SPJ (Storage-Partitioned Join) shims ported to 4.2.
- **CI wiring**: Spark 4.2 resource install + change detection in the
workflow util script.
- Canonical ASF license headers across `shims/spark42`.
There is **no unit-test module** in this PR (that is a follow-up); this
change is the profile + shim layer that must compile and integrate cleanly.
## How was this patch tested?
Validated via a full **Velox Backend (x86)** CI run on a personal fork —
**41/41 jobs green** (build-fast-build-test, build-native-lib-centos-8,
build-cudf, cpp-test-udf, and all `spark-test-spark34/35/40/41` + `tpc-test-*`
lanes). The branch is rebased on the latest `main` (0 commits behind).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
