[
https://issues.apache.org/jira/browse/SPARK-60448?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
ASF GitHub Bot updated SPARK-60448:
-----------------------------------
Labels: pull-request-available (was: )
> Apply filter conditions on a storage filter's key columns in the Parquet
> reader
> -------------------------------------------------------------------------------
>
> Key: SPARK-60448
> URL: https://issues.apache.org/jira/browse/SPARK-60448
> Project: Spark
> Issue Type: Improvement
> Components: SQL
> Affects Versions: 5.0.0
> Reporter: Victor Sunderland
> Priority: Major
> Labels: pull-request-available
>
> A follow-up to SPARK-59620.
> With {{spark.sql.parquet.storageFilterPushdown.enabled}} on, the vectorized
> Parquet reader reads the join key column first, checks each row against the
> join's runtime bloom filter, and reads the other columns only for the rows
> that pass. Any other condition on that same key column, such as {{k % 50 =
> 0}}, is still evaluated only in the Filter after the scan. The reader
> therefore reads the other columns for rows that are dropped right afterwards.
> This proposes offering such conditions to the reader along with the bloom
> filter. The reader already holds the key column, so evaluating one more
> condition on it adds no I/O, and the rows it rejects never have their other
> columns read. Concretely:
> * A new hook, {{FileFormat.supportsStorageFilterOnKeys(expr)}}, false by
> default. Parquet accepts any condition it can evaluate as is (no subqueries,
> no unevaluable expressions such as Python UDFs).
> * {{FileSourceStrategy}} offers a conjunct that references only the bloom
> filter's key columns after the bloom filter. {{IsNotNull}} is not offered,
> since the post-scan Filter needs it as is to mark the key column non-nullable.
> * The reader evaluates these conditions only on the rows the bloom filter
> keeps, so that subexpression elimination does not compute what they share on
> every row.
> This does not help a plain inner join, where Spark already infers such a
> condition onto the other side before it builds the bloom filter. It helps
> when the condition is not in the bloom filter, for example when the fact
> table is on the nullable side of a LEFT OUTER JOIN and the condition is
> inside a derived table, or for rows that pass only as bloom filter false
> positives. It saves the most with wide rows in small pages, where removing
> rows empties whole pages that the reader can then skip.
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
