schenksj opened a new issue, #4809: URL: https://github.com/apache/datafusion-comet/issues/4809
Follow-up to #4807 (runtime dynamic filter pushdown for native hash joins). Blocked on the native Delta scan contrib landing (tracking PR #4366). The generic phase of #4807 applies the join's `DynamicFilterPhysicalExpr` as a row-level filter above the probe-side scan, which the Delta contrib's `DeltaKernelScanExec` inherits with no format-specific work. This issue covers the IO-level upgrade so the filter also prunes inside the delta-kernel-rs read. ## Why Delta is well positioned `DeltaKernelScanExec` (contrib, PR #4366) is a kernel-read operator like `IcebergScanExec` — not a `ParquetSource` — so it does not get DataFusion's optimizer-driven scan pushdown. However, the contrib already ships a Spark -> delta-kernel predicate converter (`contrib/delta/native/src/predicate.rs`) used for static data skipping. The IO-level dynamic filter upgrade can reuse it rather than writing a new conversion layer. ## Proposed shape 1. Snapshot the populated `DynamicFilterPhysicalExpr` at first poll of the probe-side scan stream (join build completes before the probe stream is polled; may require deferring kernel-read construction into the stream, mirroring the Iceberg follow-up). 2. Convert the convertible conjuncts (min/max bounds always; `InList` when present) through the existing `predicate.rs` converter and attach to the kernel scan's physical predicate. The `HashTableLookupExpr` strategy is not convertible; keep bounds only (row-level wrapper from #4807 still applies the full filter). 3. Verify how much row-group pruning delta-kernel-rs's default-engine Parquet read actually performs with an executor-side predicate — this bounds the payoff and should be measured before/while implementing. 4. File-level pruning is out of scope: the file list is resolved driver-side against Delta log stats. ## Decision input `dynamic_filter_rows_pruned` / selectivity metrics from #4807 on Delta star joins. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
