lxc512157407 opened a new pull request, #24821:
URL: https://github.com/apache/datafusion/pull/24821
## Which issue does this PR close?
<!--
We generally require a GitHub issue to be filed for all bug fixes and
enhancements and this helps us generate change logs for our releases. You can
link an issue to this PR using the syntax "Closes #123" .
-->
N/A (new behavior, no existing issue filed yet)
## Rationale for this change
Join planning inserts null-rejecting `IS NOT NULL` filters on nullable join
keys (`filter_null_join_keys`), which are then pushed down to scans. When the
underlying data contains no NULLs (the overwhelmingly common case for join
keys, e.g. TPC-H primary/foreign keys), these filters pass 100% of rows —
we measured `selectivity = 100% (6.00M/6.00M)` on TPC-H SF=10 — yet every
input batch still pays a full predicate evaluation plus a full batch copy.
`FilterExec::statistics_helper` already derives, in the forward direction,
that a surviving `IS NOT NULL` conjunct implies `null_count = Exact(0)` for
the output. This PR adds the statistics-driven **inverse**: when the *input*
statistics already prove `null_count = Exact(0)` for a column, a bare
`Column IS NOT NULL` conjunct is vacuously true and is dropped from the
predicate at `FilterExecBuilder::build` time. If all conjuncts are dropped,
the predicate becomes `lit(true)`.
## What changes are included in this PR?
- `simplify_not_null_conjuncts`: at `FilterExecBuilder::build`, split the
predicate into conjunctions; drop bare `Column IS NOT NULL` conjuncts whose
column reports `null_count == Exact(0)` in the input statistics
(`StatisticsContext::compute`, all-partitions). Non-provable cases
(Absent/Inexact) leave the predicate untouched.
- 3 unit tests: conjunct dropped when provable; whole predicate → `lit(true)`
when everything is dropped; predicate unchanged when `null_count` is not
`Exact`.
## Are these changes tested?
Yes — unit tests in `datafusion/physical-plan/src/filter.rs`; the existing
171 filter tests pass unchanged (statistics with `Absent` null_count are
unaffected).
## Are there any user-facing changes?
No API changes. Behavior: plans over sources that report exact zero
null-count statistics now execute fewer no-op filter evaluations. Sources
that do not report statistics behave exactly as before.
<!--**
To ensure clarity, PRs are reviewed against
https://github.com/apache/datafusion/blob/main/docs/asf-ci-testing.md#source-code-considerations
-->
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
