lxc512157407 commented on PR #24821: URL: https://github.com/apache/datafusion/pull/24821#issuecomment-5662939242
## Benchmark correction & approach change My earlier comment reported a 15% speedup on the single-table case. That number was accurate but the conclusion was **incomplete** — follow-up join benchmarks showed the build-time approach regressed joins by ~45%. This is now fixed with a different approach. ### Why build-time simplification was wrong Dropping the predicate at FilterExecBuilder::build() time (when input stats report null_count == Exact(0)) turned the predicate into lit(true). A physical optimizer rule (FilterPushdown, empty-conjunct case) then deleted the whole FilterExec node. The side effect: FilterExecStream's implicit batch coalescing (4096-row scan batches → 8192-row batches) disappeared, and HashJoin received 2x more, 2x smaller batches — 488 → 976 input batches in my benchmark — a measured 45% join slowdown. ### New approach: per-batch fast path in FilterExecStream When the predicate is exactly a bare "col IS NOT NULL" and the incoming batch's column has null_count() == 0 (an O(1) cached value in arrow), skip mask evaluation and filter_record_batch entirely and push the batch into the coalescer unchanged: - Plan shape is untouched — the FilterExec node stays, so repartition/coalescing decisions downstream are identical to main - The check is per-batch: a batch that actually contains NULLs takes the normal evaluation path - Deliberately narrow (whole-predicate match only, no projection): larger predicates fall through to the existing path ### Benchmark (4M rows, MemTable, nullable column with zero NULLs) | Query | main (ms) | PR (ms) | |-------|-----------|---------| | sum(id) | 0.68 | 0.67 | | sum(id) WHERE id IS NOT NULL | 0.94 | 0.75 (~20% faster) | Join queries now show identical physical plans to main (verified via EXPLAIN ANALYZE: same RepartitionExec/CoalescePartitionsExec chain, same 488 input batches into HashJoin), so the earlier join regression is gone while the single-query gain is kept. Unit tests added for both paths (null-free batch passes through; batch containing NULLs filters normally). -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
