manoj-ragupathy opened a new pull request, #13020:
URL: https://github.com/apache/gluten/pull/13020
## What changes are proposed in this pull request?
Pre-work for Spark 4.2.x support (#12569), following the pattern of #11313
in the Spark 4.1 bring-up: land the small, self-contained compatibility fixes
first so the shim-layer and unit-test PRs that follow stay reviewable.
Every change here is **version-agnostic** — this PR adds no `spark42` paths
and no `spark-4.2` profile. It only reshapes code that Spark 4.2 would
otherwise break, in a way that works identically on Spark 3.4 through 4.1.
| Change | Why Spark 4.2 needs it |
|---|---|
| Move `doPostDriverMetrics` out of `BatchScanExecTransformer` into each
`BatchScanExecShim` | 4.2 moved `postDriverMetrics` to
`SupportsCustomDriverMetrics` and made the reported task metrics an explicit
argument, so the call site has to differ per version |
| Add `getSampleSeed` to `SparkShims` | `SampleExec.seed` became
`Option[Long]`, resolved lazily via `resolvedSeed` |
| Add `isKeyGroupedPartitioning` to `SparkShims` | `KeyGroupedPartitioning`
was renamed to `KeyedPartitioning` |
| Match on type instead of position in `ConverterUtils` and
`GlutenNoopWriterRule` | `CharType` gained `collation`; `AppendDataExec` and
`OverwriteByExpressionExec` gained `tableName` and `transaction`, silently
breaking positional extractor patterns |
| Make `GlutenQueryTest` extend Spark's `QueryTest` | 4.2 hoisted
`checkAnswer`, `checkDataset`, `assertCached` and friends into a new
`QueryTestBase` trait that `SharedSparkSession` now mixes in; because
`GlutenQueryTest` declared its own copies, suites mixing both inherited
conflicting definitions. Extending `QueryTest` (an abstract class up to 4.1, a
trait in 4.2) makes Gluten's versions genuine overrides on every supported
version |
| Add `getBinaryView` stub to the `SpecializedGetters` compatibility shims |
`SpecializedGetters` gained `getBinaryView` in 4.2. The return type is
`Nothing` on purpose, matching the existing
`getVariant`/`getGeography`/`getGeometry` stubs, so one definition works across
the whole 3.x/4.x matrix |
| Use `ByteBuffer.allocateDirect` in `OnHeapFileSystemTest` | Spark 4.2
bumps Netty to 4.2.13, which removed
`PlatformDependent.allocateDirectNoCleaner` |
| Ignore a `datasketches` duplicate class in `package/pom.xml` | Spark 4.2's
`spark-catalyst` ships a patched copy of `ResourceImpl`, tripping the
ban-duplicate-classes enforcer. Both artifacts are `provided`, so nothing extra
is packaged |
The two new `SparkShims` methods are implemented in all existing version
shims (`spark34`, `spark35`, `spark40`, `spark41`), keeping `gluten-substrait`
and `backends-velox` version-agnostic.
Part of #12569.
## How was this patch tested?
No behaviour change is intended on any currently supported Spark version, so
this relies on existing test coverage.
Compile-verified on all four supported profiles:
```bash
for cfg in "3.4 2.12" "3.5 2.12" "4.0 2.13" "4.1 2.13"; do
set -- $cfg
./build/mvn -Pspark-$1 -Pscala-$2 -Pjava-17 -Pbackends-velox -Pspark-ut \
clean test-compile -DskipTests
done
```
All four report `BUILD SUCCESS`. The `GlutenQueryTest` change is the
highest-risk one here, since it affects every suite that mixes it with
`SharedSparkSession`; it was verified on Spark 3.4 (the oldest supported
version) as well as 4.1.
## Was this patch authored or co-authored using generative AI tooling?
Generated-by: GitHub Copilot CLI (Claude Opus 5)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
