LuciferYang commented on issue #9883:
URL: https://github.com/apache/paimon/issues/9883#issuecomment-5715345869
## Follow-up design for the partially-evaluable case
#9909 fixed the fully-unevaluable case: when the scalar global index cannot
evaluate the predicate at all, `scalarMatchedRows` returns `null` and the
ranges are routed to the raw search where the exact filter decides. This issue
stays open for the partially-evaluable case.
### The remaining problem
When a WHERE filter has one conjunct the index can evaluate and one it
cannot, `GlobalIndexEvaluator` silently drops the unevaluable conjunct and
returns a non-null result that is a superset of the true matches. The vector
index path trusts that bitmap as the exact include set (it has no final-read
filter), so the top-K is computed over the superset: rows that fail the dropped
conjunct can take top-K slots, and truly-matching rows ranked below them are
lost.
Two shapes, both reproducible:
- different fields: `a = 1 AND tags IS NOT NULL`, where `IS NOT NULL` on a
multivalue-indexed array is unsupported;
- same field: `id >= 0 AND CAST(id AS BIGINT) > 0`, where the CAST conjunct
is unsupported.
### Field coverage is not sufficient
An earlier attempt gated "keep the index result" on whether every field the
filter references contributed (`contributingFieldIds`). That is not enough: the
same-field example has `contributingFieldIds = {id}` covering every referenced
field, yet the result is still a superset because a second conjunct on `id` was
dropped. Exactness is a property of the whole predicate evaluation, not of
field coverage.
### Where it lives (per engine)
The gap is at the evaluator level and is engine-agnostic. Anything that
reaches the vector-search `filter` is affected, and both example predicates do
reach it:
- Spark pushes convertible predicates into the scan;
`SparkV2FilterConverter` converts both `CAST(id AS BIGINT) > 0` (a cast
transform) and `array IS NOT NULL`, so they reach the evaluator.
- Flink's `vector_search` procedure parses the whole `where` string into the
Paimon predicate; supported functions reach the evaluator, and an inexpressible
one such as CAST fails the call rather than silently dropping.
- Local and batch reads pass the full predicate directly.
One case is out of scope here: a genuinely non-convertible Spark predicate
(a UDF, a column-to-column comparison, an unresolvable cast) stays as a Spark
residual, never reaches Paimon, and Spark post-filters the already-truncated
top-K, which cannot refill displaced rows. That is a separate Spark-boundary
concern, not the "scalar index cannot evaluate" problem tracked here; I will
file it separately.
### Proposed design
1. **Exactness signal in the evaluator.** Add an `exact` flag sourced from
the reader/result (default exact; an approximate reader reports inexact),
propagated bottom-up in `combineResults`: an AND is exact only if no child was
dropped and every contributing child is exact; an OR is exact if all children
are exact (it already requires all present); an empty result is exact. A
boolean is sufficient. Sourcing it from the reader, rather than inferring it
from "the reader returned a present result", keeps the guarantee under future
approximate or ANN-style readers.
2. **Remediation, gated on the flag.** When the result is exact, use it as
today. When it is inexact, keep the index (ANN) path but re-check: apply the
supported part as a pre-filter, over-retrieve more than K candidates (the
existing refine-factor path already over-retrieves), re-apply the exact filter
on the candidate rows (the candidate read path can already read filter columns
and run the exact predicate), take the top-K, and iteratively retrieve more if
fewer than K survive. Fall back to the raw search when the residual is
selective enough that refilling approaches a full scan. This keeps the index
fast path for a selective supported conjunct and stays correct.
All four read paths (local, batch, Spark, Flink) inherit this from the
shared base, so it is a single fix.
### Alternatives considered
- Route every inexact predicate to a full raw search, like the
fully-unevaluable case. Simplest, but it loses the index acceleration for a
selective supported conjunct, so it is kept only as the safety valve above.
- Gate on `contributingFieldIds`. Insufficient, as shown above.
Feedback welcome, especially on the `exact` flag versus a residual
predicate, and on candidate re-check versus raw fallback as the default.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
