akshaytayal opened a new pull request, #13520:
URL: https://github.com/apache/gluten/pull/13520
## What changes are proposed in this pull request?
Part of #13517. Adds the Spark 4.3 unit test module, `gluten-ut/spark43`,
and its CI jobs, like #13163 for Spark 4.2.
**Depends on #13519** (which depends on #13518). Until they merge, this
branch includes their commits; this PR's own changes are the last 3 commits.
Following #11352, they keep the history of the moved files and should be merged
with **rebase**, not squash:
1. `Spark: Move 4.2 UT as 4.3 UT`: `git mv gluten-ut/spark42
gluten-ut/spark43` (renames only).
2. `Spark: Copy back 4.3 UT as 4.2 UT`: restores `gluten-ut/spark42`
unchanged.
3. `Spark: Initial support for 4.3.0 UT`: the Spark 4.3 changes:
- **Wiring:** `spark-4.3` profile in `gluten-ut/pom.xml`, artifactId
`gluten-ut-spark43`, and the spark43 TPC-DS plan stability golden files
location.
- **New suites:** Gluten wrappers for 91 Spark test suites added in Spark
4.3, registered in `VeloxTestSettings`.
- **Spark 4.3 API changes in suites**, for example the
storage-partitioned join conf renamed to
`V2_BUCKETING_ALLOW_KEYS_SUBSET_OF_PARTITION_KEYS`.
- **Failing Spark 4.3 UTs are disabled, not modified**, each tagged
`GLUTEN-13517`: 157 `.exclude(...)` entries in `VeloxTestSettings`, 22 `.sql`
files removed from the `VeloxSQLQueryTestSettings` allowlist, and disabled
suites for whole-suite aborts. They will be grouped by root cause into
follow-up issues under #13517.
- **backends-velox and gluten-core tests, Spark 4.3 only:**
- `VeloxAggregateFunctionsSuite` accepts the Complete mode aggregate
that Spark 4.3 creates by combining adjacent Partial and Final aggregates
(SPARK-43317).
- `ScalarFunctionsValidateSuite` runs the higher-order function tests
with on-heap column vectors. Spark 4.3's new codegen for higher-order functions
reads null elements of off-heap Parquet dictionary vectors and fails in vanilla
Spark.
- Suites that use `GlutenShuffleManager` (`GlutenShuffleManagerSuite`,
`ShuffleManagerRouterCacheSuite`, `VeloxTPCHV1GlutenShuffleManagerSuite`) are
skipped: Spark 4.3 requires a `BlockingShuffleManager` (SPARK-58185).
- **CI:** `spark-test-spark43` (3 groups) and `spark-test-spark43-slow`
(extended, slow-hive), like the Spark 4.2 jobs but without `-Pdelta`. Until the
CI images provide `/opt/shims/spark43` (after #13519 merges), the jobs install
the Spark 4.3 resources with `install-spark-resources.sh`.
## How was this patch tested?
The new `spark-test-spark43` and `spark-test-spark43-slow` jobs. Built
locally against Spark 4.3.0, including all test sources. The disabled set was
CI-triaged earlier against Spark 4.3.0-RC2 in a private CI run, where all Spark
4.3 UT jobs passed.
## Was this patch authored or co-authored using generative AI tooling?
Generated-by: GitHub Copilot CLI (Claude Opus 5.5)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
