JingsongLi commented on PR #9909:
URL: https://github.com/apache/paimon/pull/9909#issuecomment-5709427246
One more thing, on scope — this fixes the first shape of #9883, but not the
second, and none of the new code paths fire for it.
#9883 lists both shapes explicitly:
> 2. The filter is a partially-evaluable `AND`, e.g. `a = 1 AND tags IS NOT
NULL` where `tags IS NOT NULL` is unsupported.
with the symptom "the top-K is polluted by rows that do not match the filter
and matching rows are displaced out of it".
That shape still reproduces here:
- `GlobalIndexEvaluator#combineResults` silently drops absent `AND`
children, so evaluation *succeeds* and returns a superset matching `a = 1`
alone, with `contributingFieldIds = {a}`.
- `demoteUncoverableIndexSplits` only fires when `scalarMatchedRows` returns
`null`; a non-null superset is taken as exact, and the index read path has no
final-read filter to correct it.
- No raw split covers the range either: `DataEvolutionVectorScan` derives
the filter's raw ranges from field-level coverage over
`collectFieldIds(rowType, filter)`. Both `a` and `tags` have index files for
the range, so `unindexedRanges` is empty and no `RawVectorSearchSplit` is added.
The PR body is explicit that this is deliberate — "keeps the index fast path
for a partially-evaluable predicate ... only the fully-unevaluable case is
demoted" — but it also says `close #9883`, and #9883 explicitly covers both
shapes. I would either keep #9883 open as the tracker for shape 2 (the new test
only covers shape 1), or widen the demotion criterion to "the contributing
fields must cover every field the filter references" — which is exactly what
#9884 did, and exactly why it routed this `AND` shape to the raw search. If
that ANN regression is not acceptable, the performance vs. correctness call
should be made explicitly, and shape 2 probably needs a different mechanism
than demotion (e.g. over-fetch plus a residual re-check of the candidates, in
the spirit of the existing refine path) rather than staying silently inexact.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
