JingsongLi commented on code in PR #11254:
URL: https://github.com/apache/paimon/pull/11254#discussion_r4623638367
##########
paimon-python/pypaimon/execution/python/read/search/vector_search_read.py:
##########
@@ -848,6 +871,67 @@ def _score_raw_batch_queries(self, batches, metric):
return [_scored_result(heap) for heap in heaps]
+def _open_raw_batch_reader(table_read: "TableRead", splits: "List[Split]") ->
"_ClosableArrowBatchReader":
+ from pypaimon.read.read_support import _ClosableArrowBatchReader
+
+ reader, source = table_read._new_arrow_batch_reader(splits)
+ return _ClosableArrowBatchReader(reader, source)
+
+
+def _parallel_raw_batches(
+ table_read: "TableRead", splits: "List[Split]", workers: int,
+) -> "_BatchStream":
+ """Yield ready batches, with at most one outstanding read per split
group."""
+ # Readers may be suspended after yielding. Join all in-flight reads
+ # before closing their iterators, including when scoring fails.
+ with ExitStack() as streams, ThreadPoolExecutor(max_workers=workers) as
executor:
+ pending: "Dict[Future[Optional[pa.RecordBatch]],
Iterator[pa.RecordBatch]]" = {}
+ try:
+ for i in range(workers):
+ group = splits[i::workers]
+ batches = _open_prefetched_batches(table_read, group, streams)
+ pending[executor.submit(next, batches, None)] = batches
+ while pending:
+ completed = wait(pending, return_when=FIRST_COMPLETED).done
Review Comment:
[P2] Keep local Top-K stable when raw vectors contain NaN
The old materializer scored splits in plan order. Scoring batches as soon as
they are ready exposes the heap to non-total ordering for NaN. For `limit=1`, a
NaN score pushed first cannot be replaced by a later finite score. I committed
`[1.0, 0.0]` and `[NaN, 0.0]` to two partitions of an unindexed Data Evolution
Parquet table, then ran local L2 search for `[1.0, 0.0]` at
`read.parallelism=2`. Delaying only the finite split returned the NaN row on
this head. Delaying only the NaN split returned the exact match. Master
returned the exact match with either delay because it scored in plan order.
Please define a deterministic non-finite score policy before inserting into the
heap, and test both completion orders. For example, consistently rank NaN after
finite scores with the same row-ID tie break.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]