EdisonWang created SPARK-59020:
----------------------------------
Summary: Support DSv2 Parquet shredded Variant predicate pushdown
Key: SPARK-59020
URL: https://issues.apache.org/jira/browse/SPARK-59020
Project: Spark
Issue Type: Improvement
Components: SQL
Affects Versions: 5.0.0
Reporter: EdisonWang
DSv2 Parquet filter pushdown currently runs before Variant extraction
pushdown. Because of that ordering, filters on variant_get expressions are not
translated into Parquet filters after the Variant extraction rewrite exposes the
corresponding shredded fields.
This change adds a narrow internal pushdown path for rewritten Variant
predicate filters. V2 scan pushdown translates eligible rewritten filters after
Variant extraction pushdown, ParquetScanBuilder records the accepted filters,
and ParquetPartitionReaderFactory passes them to the existing ParquetFilters
shredded Variant logic when the read schema contains shredded Variant metadata.
The original Spark filters remain above the scan, so this only improves
Parquet row-group skipping and does not change query results.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
