JingsongLi commented on code in PR #8459:
URL: https://github.com/apache/paimon/pull/8459#discussion_r3523222752
##########
paimon-tantivy/paimon-tantivy-index/src/main/java/org/apache/paimon/tantivy/index/TantivyFullTextGlobalIndexReader.java:
##########
@@ -96,19 +98,23 @@ public CompletableFuture<Optional<ScoredGlobalIndexResult>>
visitFullTextSearch(
SearchResult result =
borrowed.searcher.searchJson(
fullTextSearch.queryJson(),
fullTextSearch.limit());
Review Comment:
Fixed in 56df286b8: TantivySearcher now exposes numDocs(), and the Java
full-text reader uses that full-shard doc count as the search limit whenever
includeRowIds is present, then applies the include bitmap and final topK. Also
added a regression test where limit=1 would previously miss a lower-ranked
included row after higher-ranked excluded rows.
##########
paimon-python/pypaimon/globalindex/tantivy/tantivy_full_text_global_index_reader.py:
##########
@@ -346,8 +346,22 @@ def visit_full_text_search(self, full_text_search):
import tantivy
+ include_row_ids = full_text_search.include_row_ids
+ search_limit = limit
+ if include_row_ids is not None and not include_row_ids.is_empty():
+ search_limit = max(limit, include_row_ids.cardinality())
Review Comment:
Fixed in 56df286b8: the Python Tantivy reader no longer uses
include_row_ids.cardinality() as the upper bound. For include_row_ids it
requires/searches by the shard doc count from the PyTantivy searcher, then
filters and applies final top_k. The Python regression test now simulates two
excluded higher-ranked hits before the included hit.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
