JingsongLi commented on PR #9855: URL: https://github.com/apache/paimon/pull/9855#issuecomment-5711335427
### 1. Preserve exact filtering before top-k in full mode `DataEvolutionFullTextRead.java:205` The result of `scanner.scan(filter)` is not necessarily an exact match set. For example, BTree's `Contains`, `EndsWith`, and `Like` implementations return all non-null rows as candidates. This code uses that superset directly for full-text ranking, while the coverage check sees the column as fully indexed and schedules no raw refinement, even with `scalar-index.search-mode=full`. I reproduced this with a `contains(content, 'zeta')` predicate and `limit=1`: without the BTree index, the query correctly returns row 5; after adding the BTree index, it returns row 0, which does not satisfy the predicate. A subsequent Spark filter cannot recover the matching row already excluded by top-k. Could we distinguish candidate-only index results from exact matches and evaluate the residual predicate **before** full-text ranking in full mode? ### 2. Preserve the original BM25 ranking when resolving unindexed row filters `RawFullTextReadImpl.java:157` When the full-text index already covers the rows but a filter column has no scalar index, full mode routes those rows through this path. Applying the row predicate before building the temporary full-text index changes the indexed corpus—and therefore BM25 document frequencies and average document length. This is not equivalent to passing a matching-row bitmap to the existing index. I reproduced an actual top-k mismatch with the native engine: 2,000 documents, `id >= 1800`, and `limit=20`. The fallback returns row 1886, whose score in the original index is approximately **1.6786**, below the expected filtered top-20 cutoff of **1.8338**. Thus, the selected rows change, not just their reported scores. For ranges already covered by the full-text index, could we read the filter columns to produce an exact row-id bitmap and pass it through `includeRowIds`, rather than rebuilding a full-text index over the filtered documents? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
