ForVic opened a new pull request, #59486: URL: https://github.com/apache/spark/pull/59486
### What changes were proposed in this pull request? A follow-up to SPARK-59620, from its list of possible follow-ups. When a scan is offered a storage filter, it is now also offered every other conjunct that references only that filter's key columns. The reader reads those columns in phase 1 anyway, so the conjunct adds no read. It only narrows the surviving rows, so phase 2 reads fewer pages. - `FileFormat.supportsStorageFilterOnKeys(expr)`, default false. `ParquetFileFormat` accepts a conjunct it can evaluate as it is, meaning nothing unevaluable and no subquery. - `FileSourceStrategy.storageFiltersFor` appends these conjuncts after the accepted storage filters, so the reader evaluates them only on the rows the bloom keeps. The reader is unchanged. It already evaluates its storage filters as one conjunction, and SPARK-60034 already skips every offered conjunct after the scan on the rows the reader checked. ### Why are the changes needed? A bloom keeps every row whose key it holds. A conjunct on the same key that min/max statistics cannot prune, e.g. `k % 50 = 0`, is applied today only after the scan. Phase 2 therefore reads the payload pages of rows that conjunct then drops. The shapes where this matters are narrow. For an inner join, the optimizer infers such a conjunct onto the bloom's creation side, so the bloom already excludes those keys and this change only drops its false positives. It matters where the bloom does not reflect the conjunct. One example is a fact table on the null-supplying side of a LEFT OUTER join, with the conjunct in a subquery. ### Does this PR introduce _any_ user-facing change? Only with `spark.sql.parquet.storageFilterPushdown.enabled` on, which is off by default. Such conjuncts then appear in the scan's `StorageFilters` and the reader applies them. The conf doc gains one sentence. ### How was this patch tested? Two new tests in `ParquetStorageFilterSuite`: - The planner offers a key-only conjunct after the bloom. It offers none over another column and none without a bloom. - With the conjunct, the reader keeps exactly the rows the query returns. `ParquetStorageFilterSuite`, `DataFrameJoinSuite`, `SubquerySuite` and `InjectRuntimeFilterSuite` pass. **Benchmark** (not part of this PR) The benchmark scans a fact table under a bloom built from a key range, plus the conjunct `k % 50 = 0`. It has three arms: feature off, bloom only (master), and bloom plus key conjunct (this PR). - Wide table: 50k rows of a long key and a 2 kB value, in 1,500-row row groups. - Narrow table: 1.4M rows of a long key and eight longs, in 42,000-row row groups. - Both are about 100 MB with no dictionary, written as `StorageFilterPushdownBenchmark` writes them. - Run with `local[1]` on JDK 21, on a Graviton r8g.4xlarge shared with other jobs, so timings vary by ±20–50 ms. Rows and bytes are exact. Times are the best of 5. | Case | Rows out, bloom only → + conjunct | Bytes avoided, bloom only → + conjunct | Off | Bloom only | + Key conjunct | |---|---|---|---|---|---| | Should help: 2 kB rows, 8 kB pages, bloom keeps 10% clustered | 5,000 → 100 | 88 → 97 MB | 299 ms | 146 ms | 118 ms | | Should help: 2 kB rows, 8 kB pages, bloom keeps 50% clustered | 25,000 → 500 | 49 → 94 MB | 211 ms | 168 ms | 121 ms | | Should not help: 2 kB rows, default pages, bloom keeps 10% clustered | 5,000 → 100 | 88 → 88 MB | 125 ms | 102 ms | 103 ms | | Should not help: eight longs, default pages, bloom keeps 5% scattered | 70,166 → 1,375 | 0 → 0 MB | 211 ms | 215 ms | 211 ms | | Should not help: bloom already holds only matching keys (inner join) | 100 → 100 | 97 → 97 MB | 200 ms | 149 ms | 138 ms | It helps where the bloom keeps rows whose payload pages the conjunct then frees, which takes few rows per page. Where every page still holds a survivor, or the bloom already reflects the conjunct, the read takes the same bytes plus one more predicate per bloom survivor, and the difference is within noise here. ### Was this patch authored or co-authored using generative AI tooling? Generated-by: Claude Code (Claude Opus 5.5) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
