leaves12138 commented on code in PR #8459:
URL: https://github.com/apache/paimon/pull/8459#discussion_r3523212297
##########
paimon-python/pypaimon/globalindex/tantivy/tantivy_full_text_global_index_reader.py:
##########
@@ -346,8 +346,22 @@ def visit_full_text_search(self, full_text_search):
import tantivy
+ include_row_ids = full_text_search.include_row_ids
+ search_limit = limit
+ if include_row_ids is not None and not include_row_ids.is_empty():
+ search_limit = max(limit, include_row_ids.cardinality())
Review Comment:
`max(limit, include_row_ids.cardinality())` is not a safe upper bound if
more than `cardinality` excluded documents rank ahead of the included
documents. It can return fewer than `limit` included matches even though more
matching included rows exist. This should use the same strategy as the Java
side: push the row-id filter into Tantivy, or search the full shard / another
proven upper bound before applying the bitmap filter.
##########
paimon-tantivy/paimon-tantivy-index/src/main/java/org/apache/paimon/tantivy/index/TantivyFullTextGlobalIndexReader.java:
##########
@@ -96,19 +98,23 @@ public CompletableFuture<Optional<ScoredGlobalIndexResult>>
visitFullTextSearch(
SearchResult result =
borrowed.searcher.searchJson(
fullTextSearch.queryJson(),
fullTextSearch.limit());
Review Comment:
Filtering after `searchJson(..., fullTextSearch.limit())` can under-return
when `includeRowIds` is set but excluded/deleted rows occupy the top N. The
high-level `FullTextReadImpl` path currently passes
`candidateLimit(rowRangeStart, rowRangeEnd)`, so the new deleted-row flow is OK
there, but `FullTextSearch.withIncludeRowIds` is now part of the reader API and
direct calls can still miss included hits. Could we either push the row-id
predicate into Tantivy, or search the whole shard / another proven upper bound
when `includeRowIds != null`? A test where the best hits are excluded and
lower-ranked included rows should catch this.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
