JingsongLi commented on PR #9909:
URL: https://github.com/apache/paimon/pull/9909#issuecomment-5710566215
The fallback for fully unevaluable predicates looks good. However, this does
not fully resolve #9883, so please remove `close #9883` and keep the partially
evaluable case tracked.
I reproduced the remaining issue on `073d0772`: with four rows ordered by
increasing distance and only row 0 having `tags = NULL`, a search with `id >= 0
AND tags IS NOT NULL` and `limit = 2` returns `[0, 1]` instead of `[1, 2]`. The
scalar evaluator drops the unsupported conjunct, and `prepareSplits` treats the
resulting non-null bitmap as sufficient to stay on the index path. This is an
existing correctness gap, not a new regression introduced by this PR.
**One additional point for the follow-up: checking contributing field IDs is
not sufficient to establish exactness.** I also reproduced the problem with:
```sql
id >= 0 AND CAST(id AS BIGINT) > 0
```
The BTree index evaluates the first conjunct, while the CAST predicate is
unsupported. Here, `contributingFieldIds = {id}` covers every referenced field,
yet the search still returns `[0, 1]` instead of `[1, 2]`.
The evaluator needs an expression-level exactness indicator or residual
predicate, rather than just field coverage. Inexact results can then trigger
raw fallback, or candidate re-checking with additional retrieval as needed to
fill the filtered top-K. Filtering only the initial K candidates would still
miss qualifying rows displaced from that candidate set. Keeping that work
separate is reasonable for this narrowly scoped fix.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
