JingsongLi commented on code in PR #10583:
URL: https://github.com/apache/paimon/pull/10583#discussion_r4281749905
##########
paimon-python/pypaimon/globalindex/full_text/native_full_text_global_index_reader.py:
##########
@@ -203,6 +198,23 @@ def close(self) -> None:
self._stream = None
+def _search_top_k(reader, query, limit, filter_bytes=None):
+ # The native collector breaks score ties by document ordinal, whereas
+ # Python ranks ties by row ID. Read beyond the boundary until every tied
+ # candidate is available, without fetching the whole corpus otherwise.
+ maximum = 2147483647
+ fetch = min(limit + 1, maximum)
+ while True:
+ row_ids, scores = reader.search(query, limit=fetch,
filter_bytes=filter_bytes)
+ selected = DictBasedScoredIndexResult(dict(zip(row_ids,
scores))).top_k(limit)
+ getter = selected.score_getter()
+ if (len(row_ids) < fetch or len(row_ids) <= limit or fetch == maximum
+ or min(scores) < min(getter(row_id) for row_id in
selected.results())):
+ # Do not retain a score getter closing over the larger candidate
map.
+ return DictBasedScoredIndexResult({row_id: getter(row_id) for
row_id in selected.results()})
+ fetch = min(fetch * 2, maximum)
Review Comment:
Verified the fix in f49c39e59fab8fc782e205aaf88b55153e048aea. All 143
focused tests and project flake8 passed. Repeating the same 100,000-row
end-to-end tie benchmark for seven alternating runs measured 224 ms on the
original baseline and 214 ms on the new head, with identical rows and scores
(the previous revision measured 598 ms). The bounded probes address this
finding.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
