zhuqi-lucas commented on PR #22450: URL: https://github.com/apache/datafusion/pull/22450#issuecomment-4519217985
Thanks @adriangb! On the `is_complete` gate — you're right. For hash-join filters that are already complete at scan-startup, the static `PruningPredicate` at file-open has used the final value, and per-RG splitting buys nothing after that. `Inner::is_complete` is exposed; I'll tighten the gate to `is_dynamic_physical_expr(p) && !is_filter_complete(p)`, which keeps TopK and concurrent hash-join (build finishing after probe starts) on the splitting path, and lets already-complete filters fall through. Pushing in a moment. On the #22144 overlap and morsel intersection — agreed on both; will trace #22144 for a shared abstraction in a follow-up, and morsels would be the natural cross-file extension point. Have a good trip :) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
