wangzhigang1999 commented on code in PR #11254:
URL: https://github.com/apache/paimon/pull/11254#discussion_r4624226229
##########
paimon-python/pypaimon/execution/python/read/search/vector_search_read.py:
##########
@@ -848,6 +871,67 @@ def _score_raw_batch_queries(self, batches, metric):
return [_scored_result(heap) for heap in heaps]
+def _open_raw_batch_reader(table_read: "TableRead", splits: "List[Split]") ->
"_ClosableArrowBatchReader":
+ from pypaimon.read.read_support import _ClosableArrowBatchReader
+
+ reader, source = table_read._new_arrow_batch_reader(splits)
+ return _ClosableArrowBatchReader(reader, source)
+
+
+def _parallel_raw_batches(
+ table_read: "TableRead", splits: "List[Split]", workers: int,
+) -> "_BatchStream":
+ """Yield ready batches, with at most one outstanding read per split
group."""
+ # Readers may be suspended after yielding. Join all in-flight reads
+ # before closing their iterators, including when scoring fails.
+ with ExitStack() as streams, ThreadPoolExecutor(max_workers=workers) as
executor:
+ pending: "Dict[Future[Optional[pa.RecordBatch]],
Iterator[pa.RecordBatch]]" = {}
+ try:
+ for i in range(workers):
+ group = splits[i::workers]
+ batches = _open_prefetched_batches(table_read, group, streams)
+ pending[executor.submit(next, batches, None)] = batches
+ while pending:
+ completed = wait(pending, return_when=FIRST_COMPLETED).done
Review Comment:
Thanks for catching this. I’ve kept the existing NaN behavior for local and
Ray single-vector searches and made local Top-K selection deterministic.
Local search ranks NaN below every numeric score and breaks ties by smaller
row ID, including ties between NaNs. If fewer than K candidates have numeric
scores, the remaining slots can contain NaN rows with their original scores.
Top-K selection and `order_by_score()` use the same ordering. Ray still raises
an error as soon as a worker computes a NaN score, before its local Top-K can
discard that row.
I tested the same local Parquet table with two splits and forced each split
to finish first in turn. Rows 1 and 2 both scored `1.0`, and rows 0 and 3
scored NaN. Both completion orders returned row 1 for K=1 and `[1, 2, 0, 3]`
for K=4, with both NaN scores preserved. I also ran the two-split test on Ray
in both orders. A NaN score failed the query in both runs, and equal numeric
scores selected the smaller row ID in both runs.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]