Victor Sunderland created SPARK-60500:
-----------------------------------------
Summary: Offer key-column conditions holding an uncorrelated
scalar subquery to the Parquet storage filter
Key: SPARK-60500
URL: https://issues.apache.org/jira/browse/SPARK-60500
Project: Spark
Issue Type: Improvement
Components: SQL
Affects Versions: 5.0.0
Reporter: Victor Sunderland
Follow-up to SPARK-60448.
SPARK-60448 lets the Parquet reader apply the query's other conditions on a
runtime bloom filter's key columns while it reads, under
{{spark.sql.parquet.storageFilterPushdown.keyConditions.enabled}}.
{{ParquetStorageFilter.isSupportedFilterOnKeys}} rejects any condition that
contains a {{PLAN_EXPRESSION}}. So a condition holding an uncorrelated scalar
subquery, such as {{k > (SELECT max(x) FROM t)}}, is not offered.
{{FileSourceScanExec.preparedStorageFilters}} already turns every
{{execution.ScalarSubquery}} in {{storageFilters}} into a literal before the
reader sees it. That is how the bloom filter's own bytes reach the reader. So
the rule {{FileSourceStrategy}} already uses for data filters,
{{!(e.containsPattern(PLAN_EXPRESSION) &&
!e.containsPattern(SCALAR_SUBQUERY))}}, could let these conditions through as
well.
Raised by Peter Toth in
https://github.com/apache/spark/pull/59486#discussion_r4348267931
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
