lxc512157407 commented on PR #24821: URL: https://github.com/apache/datafusion/pull/24821#issuecomment-5662936193
## Benchmark correction My previous comment reported a 15% speedup on the single-table case, which is accurate but **incomplete**: ### Root cause of the earlier regression The initial implementation used **build-time simplification**: when input statistics reported , the predicate was dropped entirely at . This created a predicate, which triggered a different physical optimizer rule ( with empty conjuncts → deletes the node entirely). The side effect: the **implicit batch coalescing** inside was lost, and downstream operators (especially joins) received 2× more small batches, causing a 45% slowdown on join queries. ### The corrected approach Instead of build-time simplification, I implemented a **runtime fast path** in : col IS NOT NULL Benefits: - **No plan changes**: the node stays, so repartition/coalesce decisions are unchanged - **Per-batch**: checks (O(1) cached in arrow) at runtime, not global stats - **Single conjunct only**: deliberately narrow; larger predicates fall through to existing logic - **100% safe**: if a batch actually contains NULLs, the normal evaluation path is taken ### Benchmark results (4M rows, MemTable) | Query | main (ms) | PR (ms) | |-------|-----------|---------| | sum(id) no filter | 0.68 | 0.67 | | sum(id) WHERE id IS NOT NULL | 0.94 | **0.75 (~20% faster)** | The single-table speedup is preserved (15%→20%). More importantly, **join queries no longer regress** — the physical plan shape is identical to main. Tests updated and passing. Ready for re-review. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
