wangzhigang1999 opened a new pull request, #11254:
URL: https://github.com/apache/paimon/pull/11254

   Closes #11253
   
   ### Purpose
   
   PyPaimon's local executor currently collects all raw vectors for a single 
query before it scores the first batch. This change passes Arrow batches 
directly to the existing scorer, which updates one Top-K heap throughout the 
scan.
   
   With parallel reads, each split group has at most one read in flight and one 
batch of lookahead. The calling thread scores whichever batch is ready. If 
scoring fails, it waits for outstanding reads before closing the readers.
   
   ### Tests
   
   The new tests cover parallel Top-K results, bounded read-ahead, candidate 
filtering, and reader cleanup after errors. The regression run passed 202 tests 
and skipped one S3 checksum test that requires PyArrow 22+. Both versions 
matched the published L2 Top-10 for GIST1M queries 0, 499 and 999. 
[Dataset](https://github.com/erikbern/ann-benchmarks#data-sets)
   
   Both versions read the same unindexed GIST1M snapshot (1M rows × 960 
dimensions, 12 splits). The table shows medians for query 0 over three runs 
after one warmup per version, comparing master `d90a902d` with the patch.
   
   | Executor | Storage | Peak scoring-process RSS (MiB), before → after | 
Query time (s), before → after |
   | --- | --- | ---: | ---: |
   | Local (2 CPUs) | Local Parquet | 4,865 → 795 (-83.7%) | 12.90 → 8.57 
(-33.6%) |
   | Local (2 CPUs) | OSS | 5,158 → 683 (-86.8%) | 13.25 → 9.47 (-28.5%) |
   | Ray (2 workers) | Worker-local Parquet | 268 → 268 (0.0%) | 11.69 → 11.54 
(-1.3%) |
   | Ray (2 workers) | OSS | 282 → 279 (-1.1%) | 18.35 → 18.75 (+2.2%) |
   
   Ray workers already score one split at a time from a batch reader, and their 
peak RSS stayed close across versions. The Ray rows report the largest scoring 
worker's peak and include the time to fetch the ten hits.
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]

Reply via email to