LuciferYang opened a new pull request, #67774:
URL: https://github.com/apache/doris/pull/67774
### What problem does this PR solve?
Issue Number: close #67772
Related PR: #67771
Problem Summary:
Zone map pruning only fires when a predicate compares a column against a
constant. A predicate over two columns of the same table, `WHERE a != b` or
`WHERE a < b`, never participates, so every segment and every Parquet row group
is read even when the two columns' min/max ranges make the predicate
unsatisfiable.
Everything on the path except the leaf already supports this.
`ZoneMapEvalContext` is keyed per slot and the segment-level and Parquet
row-group-level callers already populate an entry for every slot a compound
expression references, which is why cross-column AND / OR compounds prune
today. What was missing is the comparison operator's capability gate: it
accepts one slot plus one literal and nothing else, so a two-slot comparison is
dropped before evaluation.
This PR adds the slot-vs-slot shape next to the existing slot-vs-literal one
and gives the six always-false tests a single implementation shared by both.
With the left operand's non-null values in `[lmin, lmax]` and the right
operand's in `[rmin, rmax]`, the comparison cannot be TRUE for any row when:
| operator | condition |
| --- | --- |
| `=` | `lmin > rmax \|\| rmin > lmax` |
| `!=` | `lmin == lmax && rmin == rmax && lmin == rmin` |
| `<` | `lmin >= rmax` |
| `<=` | `lmin > rmax` |
| `>` | `rmin >= lmax` |
| `>=` | `rmin > lmax` |
Substituting the degenerate range `[literal, literal]` for the right operand
reduces each row to the slot-vs-literal test already in the tree, so the
single-column path keeps its exact current behaviour. The existing tests
passing unchanged is the mechanical check on that.
`kNoMatch` means "no row in this zone can make the conjunct TRUE", not
"every row is FALSE", so NULL does not weaken these rules: a WHERE conjunct
discards FALSE and NULL alike, and the bounds summarize exactly the non-null
values. A column with no non-null value at all makes the comparison NULL on
every row and returns `kNoMatch` unconditionally.
One trap for reviewers: `can_evaluate` must not be widened. It is shared by
dictionary filtering and by `can_evaluate_equality`, and both dereference
`extract_slot_and_literal` behind a `DORIS_CHECK`, so accepting a slot-vs-slot
expression there would abort. This PR adds a separate `can_evaluate_slot_slot`
and ORs it into `can_evaluate_zonemap_filter` only; a unit test pins that down.
Two deliberate limitations: a cast around either operand is rejected,
because the bounds are raw stored values and the optimizer inserts a cast
whenever the two column types differ; and on Parquet a floating column pair
does not prune at all, because Parquet bounds omit NaN without reporting how
many were skipped, which would flip the `!=` rule.
This also fixes a bug it uncovered. The flag that carries "these bounds came
from Parquet, so a hidden NaN is possible" was only ever set in the format_v2
reader; the v1 reader assigned the slot's data type and left the flag at its
default, so the NaN guard never fired on that path. Two DOUBLE columns whose
Parquet bounds both read `[1.0, 1.0]` were then pruned by the `!=` rule even
when one of them hid a NaN, dropping a row that should have come back. The rule
now lives in one setter on `SlotZoneMap` that assigns the type and the flag
together, and all three Parquet call sites go through it.
### Release note
None
### Check List (For Author)
- Test <!-- At least one of them must be included. -->
- [X] Regression test
- [X] Unit Test
- [ ] Manual test (add detailed scripts or steps below)
- [ ] No need to test or manual test. Explain why:
- [ ] This is a refactor/code format and no logic has been changed.
- [ ] Previous test can cover this change.
- [ ] No code files have been changed.
- [ ] Other reason <!-- Add your reason? -->
Unit tests cover the six operators against an interval matrix, both fully
separated and partially overlapping, plus the guard matrix: slot missing from
the context, zone map absent, incompatible types, `pass_all`, an all-null
column on either side, the same slot on both sides, and an unknown floating NaN
count. 83 tests pass across `ExprZonemapFilterTest`,
`SegmentIteratorExprZonemapTest` and `ColumnZoneMapTest`.
The regression suite adds three loads of one segment each holding a constant
in both columns, which is the shape that forces a per-segment decision: `a !=
b` drops exactly two of the three segments and `a = b` exactly one. Every query
there runs with `enable_expr_zonemap_filter` on and off and the row counts are
compared, since the counter only shows that pruning fired.
The interval rules were also checked by mutation: seven mutants (each of
`<`, `<=`, `>`, `>=` reading the wrong end of a range, dropping the right-side
`has_not_null` guard, dropping the `rmin == rmax` term from `!=`, and making
the slot-vs-slot path bail out unconditionally) each break at least one test.
- Behavior changed:
- [ ] No.
- [X] Yes.
Two changes, both intended. Cross-column comparisons now prune at the
segment level and at the row-group level on both Parquet readers, so those
queries read less. And on the v1 Parquet reader, single-column float and double
predicates using `>`, `>=`, `!=` against a non-NaN literal, or `=` against NaN,
no longer prune: that is the NaN correctness fix above, and it was already the
behaviour on format_v2.
The benefit was measured rather than estimated. On two columns whose
per-segment ranges separate naturally (a batch-increasing column against a
fixed-range one, 20 segments) `WHERE lo < hi` skips 19 of 20 segments. On
TPC-H-shaped correlated dates, where `d2` is `d1` plus a small offset, it skips
none, for an always-false and an always-true predicate alike. The window is
narrow and depends entirely on whether the two ranges separate; TPC-H Q12 is a
regression baseline here, not a benefit demonstration.
- Does this need documentation?
- [X] No.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
