JingsongLi commented on code in PR #11254:
URL: https://github.com/apache/paimon/pull/11254#discussion_r4623638367


##########
paimon-python/pypaimon/execution/python/read/search/vector_search_read.py:
##########
@@ -848,6 +871,67 @@ def _score_raw_batch_queries(self, batches, metric):
         return [_scored_result(heap) for heap in heaps]
 
 
+def _open_raw_batch_reader(table_read: "TableRead", splits: "List[Split]") -> 
"_ClosableArrowBatchReader":
+    from pypaimon.read.read_support import _ClosableArrowBatchReader
+
+    reader, source = table_read._new_arrow_batch_reader(splits)
+    return _ClosableArrowBatchReader(reader, source)
+
+
+def _parallel_raw_batches(
+    table_read: "TableRead", splits: "List[Split]", workers: int,
+) -> "_BatchStream":
+    """Yield ready batches, with at most one outstanding read per split 
group."""
+    # Readers may be suspended after yielding. Join all in-flight reads
+    # before closing their iterators, including when scoring fails.
+    with ExitStack() as streams, ThreadPoolExecutor(max_workers=workers) as 
executor:
+        pending: "Dict[Future[Optional[pa.RecordBatch]], 
Iterator[pa.RecordBatch]]" = {}
+        try:
+            for i in range(workers):
+                group = splits[i::workers]
+                batches = _open_prefetched_batches(table_read, group, streams)
+                pending[executor.submit(next, batches, None)] = batches
+            while pending:
+                completed = wait(pending, return_when=FIRST_COMPLETED).done

Review Comment:
   [P2] Keep local Top-K stable when raw vectors contain NaN
   
   The old materializer scored splits in plan order. Scoring batches as soon as 
they are ready exposes the heap to non-total ordering for NaN. For `limit=1`, a 
NaN score pushed first cannot be replaced by a later finite score. I committed 
`[1.0, 0.0]` and `[NaN, 0.0]` to two partitions of an unindexed Data Evolution 
Parquet table, then ran local L2 search for `[1.0, 0.0]` at 
`read.parallelism=2`. Delaying only the finite split returned the NaN row on 
this head. Delaying only the NaN split returned the exact match. Master 
returned the exact match with either delay because it scored in plan order. 
Please define a deterministic non-finite score policy before inserting into the 
heap, and test both completion orders. For example, consistently rank NaN after 
finite scores with the same row-ID tie break.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]

Reply via email to