viirya opened a new pull request, #58050: URL: https://github.com/apache/spark/pull/58050
### What changes were proposed in this pull request? When a Variant column is written with shredding enabled, each extracted scalar field is stored as a typed Parquet leaf column (e.g. `v.typed_value.a.typed_value` for `$.a`) carrying min/max statistics. On the DSv1 path, `PushVariantIntoScan` rewrites `variant_get(v, '$.a', 'bigint') > 999` into a struct-field access `v.`0` > 999`. Today `ParquetFilters` cannot map `v.`0`` to a physical column, so the predicate is dropped and no row-group skipping happens for shredded-variant queries. This PR maps such predicates to the physical shredded leaf and enables row-group skipping, as a pure performance win with no behavior change. The subtlety: shredding is per-row and per-file best-effort. Values that don't fit the shredded type (overflow / type mismatch) or fields not shredded in a given file are stored in opaque untyped `value` residual columns with `typed_value` NULL. Parquet min/max excludes NULLs, so pushing the predicate on the typed leaf alone would let a row group be skipped while it still contains matching rows in a residual -- silently dropping data. (A prior attempt, #54598, did exactly this and was unsound.) To stay sound, the pushed predicate is: ``` or(leafPredicate, isNotNull(residual_0), isNotNull(residual_1), ...) ``` over the leaf's sibling `value` and every ancestor level's `value` up to the top-level `v.value`. Parquet drops a row group for an `or` only when *both* sides are droppable, and `isNotNull` (`notEq(col, null)`) is droppable only when the column is entirely NULL. So a row group is skipped only when the leaf min/max cannot match **and** every residual is entirely NULL -- i.e. the whole path is provably in the typed leaf. Otherwise the row group is kept: worst case we lose the optimization, never correctness. Details: - New internal config `spark.sql.variant.shreddedPredicatePushdown.enabled` (default off), gating the optimization. - `ParquetFilters` gains an optional `variantExtractionSchema` parameter; when set, it resolves eligible scalar object-extraction paths (`$.a`, `$.a.b`) to the physical shredded leaf and residual chain, and emits the sound predicate for `Gt/GtEq/Lt/LtEq/Eq/EqualNullSafe/In`. Array-index paths and empty/`$`/companion/placeholder paths resolve to nothing. `IsNull`/`IsNotNull` on the logical field are out of scope. - Wired on the DSv1 path (`ParquetFileFormat`). **Scope: DSv1 only.** On DSv2, variant extraction is pushed through the separate `SupportsPushDownVariantExtractions` mechanism and the filter is never rewritten into `v.`0``, so it cannot be pushed for row-group skipping. DSv2 reads remain correct (the variant filter is applied post-scan); they just don't skip row groups on shredded columns. This is noted in a code comment in `ParquetScanBuilder`. ### Why are the changes needed? Predicates on shredded Variant fields currently get no row-group skipping, so queries scan all row groups even when the shredded leaf statistics prove a group cannot match. This adds that skipping soundly, improving scan performance for selective filters on shredded Variant columns. ### Does this PR introduce _any_ user-facing change? No. The optimization is gated behind an internal config that defaults to off, and when on it only affects which row groups are read -- results are unchanged. ### How was this patch tested? New unit tests in `ParquetFilterSuite` (both `ParquetV1FilterSuite` and `ParquetV2FilterSuite`): resolver behavior for single/multi-level paths, array-index rejection, synthetic (placeholder/companion/passthrough) fields resolving to nothing, absent-field, case-insensitive matching, `In`, and that the residual `IS NOT NULL` guards appear in the produced predicate. New integration suite `VariantShreddingFilterPushdownSuite`, running across DSv1/DSv2 and vectorized/non-vectorized readers: overflow fallback (matching residual row not dropped, row group not skipped), type-mismatch fallback, file without the shredded path, residual-null happy path (asserts a row group is skipped on DSv1), and multi-level `$.a.b` (skip fires + intermediate-fallback correctness). ### Was this patch authored or co-authored using generative AI tooling? Generated-by: Claude Code (Opus 4.8) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
