zhuxiangyi commented on PR #9855: URL: https://github.com/apache/paimon/pull/9855#issuecomment-5712328189
Thanks, both confirmed (your `id >= 1800` case reproduces with the same row 1886 / 1.6786 / 1.8338). Fixed in 3882a36e7. Root cause of both: the row set must be exact before top-k, and the filter must not change the corpus the index scores against. Fix: a `FilteredRowIdReader` that reads only the filter columns of a row-id set and evaluates the predicate on the data. 1. Index answers are trusted only when `contributingFieldIds` covers every predicate field and there is no `Contains` / `EndsWith` / `Like` leaf; otherwise the candidates are refined from the data. Rows without a scalar index are decided from the data in `full` mode (excluded in `fast`, as before) and no longer unioned with the index answer. 2. Rows covered by the full-text index always stay in it; the resolved bitmap goes through `includeRowIds`. The raw path indexes every uncovered row and applies the filter as a bitmap. The scalar-unindexed raw routing and the raw pre-filter are removed. Tests: `testFullTextSearchRefinesCandidateOnlyIndexResultsBeforeTopK`, `testFullTextSearchPartiallyIndexedConjunctionIsExactBeforeTopK` (fast and full), `testUnindexedFilterInFullModeKeepsOriginalRanking` and `testRawPathAppliesFilterOverTheWholeRawCorpus` (native engine). One note: candidate refinement also runs in `fast` mode, so a `contains` filter on a BTree column reads the filter column of the candidate rows. I kept correctness over the "index only" reading of `fast`: excluding the candidates instead would silently drop the matching rows. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
