qlong opened a new pull request, #57190:
URL: https://github.com/apache/spark/pull/57190
### What changes were proposed in this pull request?
Introduce a new optimizer rule `PullOutVariantExtractions` that hoists
`variant_get` / `Cast(variant)` extractions out of three operator types
that the existing `PushVariantIntoScan` / `V2ScanRelationPushDown` rules
cannot see through:
- **Aggregate function arguments** – e.g. `max(variant_get(v, '$.price',
'int'))`:
the extraction is moved into a `Project` directly below the `Aggregate`
and the
aggregate references the resulting alias. The bare variant column is
suppressed
unless it is also needed raw (e.g. as a `GROUP BY` key), so no redundant
full-variant slot is generated.
- **Sort order keys** – e.g. `ORDER BY variant_get(v, '$.price', 'int')`:
matched as `Project → Sort`; the extraction is hoisted below the `Sort` and
the original `Project` is reproduced to prevent the alias from leaking into
the output.
- **Join conditions and projections above joins** – matched as `Project →
Join`;
extractions in both the join condition and the outer `Project` are routed
to
the owning join side. A `pushSideAliases` helper then pushes the aliases
*through* any depth of chained joins so they land in a `Project` directly
above the scan (where `PhysicalOperation` collapses them with the scan,
making
them visible to the pushdown). This is necessary because
`PhysicalOperation`
stops at a `Join` node.
A `Sort` sitting over a `Join` is handled by fusing the two cases: the
order-key aliases are pushed through the join tree, not left in a `Project`
above it.
The rule is gated by a new internal config
`spark.sql.variant.pushVariantIntoScan.pullOutExtractions` (default `true`)
and is a no-op unless `spark.sql.variant.pushVariantIntoScan` is also
enabled.
Non-variant plans are untouched.
The rule is registered as the first rule in
`SparkOptimizer.earlyScanPushDownRules`,
before `SchemaPruning` and the V2 scan pushdown rules.
### Why are the changes needed?
Before this change, a `variant_get` inside an aggregate function argument,
sort key,
or join condition caused the whole variant column to be read raw (or
shredded with a
redundant full-variant slot). For example:
```sql
SELECT name, max(variant_get(v, '$.price', 'int')) FROM T GROUP BY name
```
read the entire v column even though only the price field was needed.
After this change, Spark shreds only the requested typed fields, avoiding the
full-variant I/O.
The change improves query performance for variant referenced in aggregrate,
join, sort.
### Does this PR introduce _any_ user-facing change?
No
### How was this patch tested?
Added new units. Also run correctness tests against some known workload.
**Performance result**
The test run against tpc-ds dataset (SF=5). Run B is with the new pullout
rule enabled.
```
Compare: tpcds-flat-20260710 vs tpcds-flat-pullout-20260710 (metric:
query_median)
Run A:
/Users/qlong/opensources/variant-conformance-benchmark/results/tpcds-flat-20260710/tpcds-flat/timings-variant.csv
Run B:
/Users/qlong/opensources/variant-conformance-benchmark/results/tpcds-flat-pullout-20260710/tpcds-flat/timings-variant.csv
query median_A median_B delta_s delta_%
------ -------- -------- ------- -------
q07 7.41 4.45 +2.95 +66.3%
q12 1.64 1.66 -0.02 -1.3%
q19 2.21 2.15 +0.07 +3.1%
q26 8.90 4.39 +4.51 +102.8%
q42 5.83 3.51 +2.32 +66.0%
q52 7.42 4.55 +2.87 +63.0%
q55 5.84 3.60 +2.24 +62.3%
q63 6.91 3.78 +3.13 +82.8%
q68 7.02 4.87 +2.15 +44.2%
q73 6.55 4.27 +2.28 +53.4%
q79 6.53 5.00 +1.53 +30.7%
q98 6.62 4.55 +2.07 +45.4%
Summary: 12 queries, 12 comparable
Geo-mean delta: +48.6% (Run B faster)
Total (query_median): 72.9s vs 46.8s
```
### Was this patch authored or co-authored using generative AI tooling?
Co-authored with Claude Code
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
