JingsongLi commented on code in PR #10583:
URL: https://github.com/apache/paimon/pull/10583#discussion_r4281652047
##########
paimon-python/pypaimon/globalindex/full_text/native_full_text_global_index_reader.py:
##########
@@ -203,6 +198,23 @@ def close(self) -> None:
self._stream = None
+def _search_top_k(reader, query, limit, filter_bytes=None):
+ # The native collector breaks score ties by document ordinal, whereas
+ # Python ranks ties by row ID. Read beyond the boundary until every tied
+ # candidate is available, without fetching the whole corpus otherwise.
+ maximum = 2147483647
+ fetch = min(limit + 1, maximum)
+ while True:
+ row_ids, scores = reader.search(query, limit=fetch,
filter_bytes=filter_bytes)
+ selected = DictBasedScoredIndexResult(dict(zip(row_ids,
scores))).top_k(limit)
+ getter = selected.score_getter()
+ if (len(row_ids) < fetch or len(row_ids) <= limit or fetch == maximum
+ or min(scores) < min(getter(row_id) for row_id in
selected.results())):
+ # Do not retain a score getter closing over the larger candidate
map.
+ return DictBasedScoredIndexResult({row_id: getter(row_id) for
row_id in selected.results()})
+ fetch = min(fetch * 2, maximum)
Review Comment:
[P2] Bound repeated searches when the top-k boundary has a large tie group
Each doubling reruns the complete native query. This can make a normal-sized
shard substantially slower, beyond the documented need to eventually read all
tied candidates. With the real paimon-ftindex==0.1.0, I built one default-size
100,000-row shard containing distinct texts "paimon document" + str(i) and
queried search("paimon",
column="text").select(["id"]).with_score().order_by_score().limit(10).to_list().
All documents have the same term frequency and length, hence equal scores.
After warming both paths, seven alternating runs measured a median 223 ms on
the parent implementation and 598 ms on this head, returning exactly the same
rows and scores. The new path repeatedly rescans the same corpus as the
requested prefix grows.
Please cap the number of tie-expansion probes and fall back to a known
shard/corpus bound, or use a native collector that breaks ties by row ID, and
cover this tie-heavy case in performance validation.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
