LuciferYang opened a new pull request, #12522:
URL: https://github.com/apache/gluten/pull/12522
### What changes were proposed in this pull request?
Spark 3.2 support was removed in prior PRs (#11351, #11687, #11731, #11887);
currently supported versions are Spark 3.3, 3.4, 3.5, 4.0, 4.1. A few symbols
that existed only as pre-Spark-3.3 shims are still around and dead code today.
1. `GlutenPlan.SupportsRowBasedCompatible` trait
Introduced to provide `def supportsRowBased(): Boolean` for Spark < 3.3
where `SparkPlan.supportsRowBased` did not exist yet. The default body was
`throw new GlutenException("Illegal state: The method is not expected to be
called")`. On Spark 3.3+, `SparkPlan.supportsRowBased` is native and every
concrete `GlutenPlan` / `ColumnarInputAdapter` overrides it directly, so the
trait's default was already unreachable. Drop the trait and its two mixin sites.
2. `SparkVersionUtil.gteSpark33`
Always true after Spark 3.2 was dropped. Also drop the single caller
guard in `canPropagateConvention` (Transitions), which no longer needs to skip
`UnionExec` on Spark 3.2. `eqSpark33` and `comparedWithSpark33` are kept: they
distinguish Spark 3.3 from 3.4+ (different `TaskContextImpl` ctor signature and
different write planning API), which is unrelated to the Spark 3.2 residual
concern.
3. `SparkPlanUtil.supportsRowBased` reflection
The reflection was needed on Spark 3.2 because
`SparkPlan.supportsRowBased` did not exist as a member yet; the same compiled
artifact ran on 3.2 and 3.3+ only by resolving the method reflectively at call
time. Now that Spark 3.2 is dropped, a direct call `plan.supportsRowBased`
compiles on all supported profiles and is strictly better (primitive `Boolean`
instead of boxed, no per-call `getMethod` lookup, no
`InvocationTargetException` wrapping). The 3 callers in `ConventionFunc` are on
the planning hot path.
### How was this patch tested?
Verified via compile on Spark 3.3, 3.5, and 4.1 (scala-2.13) profiles, plus
`gluten-core` and `gluten-substrait` tests on Spark 3.5. All 27 + 45 unit tests
pass. `scalastyle` and `spotless` clean.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
