Victor Sunderland created SPARK-60448:
-----------------------------------------
Summary: Apply filter conditions on a storage filter's key columns
in the Parquet reader
Key: SPARK-60448
URL: https://issues.apache.org/jira/browse/SPARK-60448
Project: Spark
Issue Type: Improvement
Components: SQL
Affects Versions: 5.0.0
Reporter: Victor Sunderland
A follow-up to SPARK-59620.
With {{spark.sql.parquet.storageFilterPushdown.enabled}} on, the vectorized
Parquet reader reads the join key column first, checks each row against the
join's runtime bloom filter, and reads the other columns only for the rows that
pass. Any other condition on that same key column, such as {{k % 50 = 0}}, is
still evaluated only in the Filter after the scan. The reader therefore reads
the other columns for rows that are dropped right afterwards.
This proposes offering such conditions to the reader along with the bloom
filter. The reader already holds the key column, so evaluating one more
condition on it adds no I/O, and the rows it rejects never have their other
columns read. Concretely:
* A new hook, {{FileFormat.supportsStorageFilterOnKeys(expr)}}, false by
default. Parquet accepts any condition it can evaluate as is (no subqueries, no
unevaluable expressions such as Python UDFs).
* {{FileSourceStrategy}} offers a conjunct that references only the bloom
filter's key columns after the bloom filter. {{IsNotNull}} is not offered,
since the post-scan Filter needs it as is to mark the key column non-nullable.
* The reader evaluates these conditions only on the rows the bloom filter
keeps, so that subexpression elimination does not compute what they share on
every row.
This does not help a plain inner join, where Spark already infers such a
condition onto the other side before it builds the bloom filter. It helps when
the condition is not in the bloom filter, for example when the fact table is on
the nullable side of a LEFT OUTER JOIN and the condition is inside a derived
table, or for rows that pass only as bloom filter false positives. It saves the
most with wide rows in small pages, where removing rows empties whole pages
that the reader can then skip.
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
