ulysses-you opened a new pull request, #18600: URL: https://github.com/apache/iceberg/pull/18600
Spark 4.2 can derive runtime filters whose values are only known at execution time (dynamic partition pruning and scalar subquery results). Filters with no V2 predicate translation, such as `part RLIKE (SELECT ...)`, never reach the scan today, so Iceberg cannot prune partitions with them. This opts `SparkRuntimeFilterableScan` into iterative runtime filtering and evaluates the received `PartitionPredicate`s against the planned tasks with the same evaluator used by static partition predicate pushdown. Predicates accepted in the first pass are reported back through `pushedPredicates()` so Spark does not derive a duplicate predicate for the same filter. Spark 4.2 only derives these predicates when every partition transform is identity. Test plan: - `TestRuntimeFiltering` (LOCAL and DISTRIBUTED): scalar subqueries over RLIKE, a UDF, and a non-first partition column; non-matching files are deleted, so the query only passes when the runtime partition predicate prunes them. - Multiple partition specs: an added identity partition field keeps the predicate evaluable for both specs. - Non-evaluable spec: after replacing an identity partition field, the runtime predicate keeps the tasks of the old spec; forcing the evaluator to treat every spec as evaluable drops matching rows. - Mixed predicates: `p1 = (subquery)` is accepted in the first pass while `coalesce(p2, 0) > (subquery)` is rejected and pushed as a partition predicate in the second. Recording every received predicate in `pushedPredicates()` makes this test fail. - Regression: `TestRuntimeFiltering`, `TestFilterPushDown`, `TestFilteredScan`, `TestSparkCopyOnWriteScanConcurrency`. --- **AI Disclosure** - Model: deepseek-flash - Platform/Tool: Claude Code - Human Oversight: [unknown - human to fill in] - Prompt Summary: Implement runtime iterative PartitionPredicate pushdown in the Iceberg Spark 4.2 module and add end-to-end tests for mixed and spec-evolved runtime predicates. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
