wangzhigang1999 opened a new pull request, #11254: URL: https://github.com/apache/paimon/pull/11254
Closes #11253 ### Purpose PyPaimon's local executor currently collects all raw vectors for a single query before it scores the first batch. This change passes Arrow batches directly to the existing scorer, which updates one Top-K heap throughout the scan. With parallel reads, each split group has at most one read in flight and one batch of lookahead. The calling thread scores whichever batch is ready. If scoring fails, it waits for outstanding reads before closing the readers. ### Tests The new tests cover parallel Top-K results, bounded read-ahead, candidate filtering, and reader cleanup after errors. The regression run passed 202 tests and skipped one S3 checksum test that requires PyArrow 22+. Both versions matched the published L2 Top-10 for GIST1M queries 0, 499 and 999. [Dataset](https://github.com/erikbern/ann-benchmarks#data-sets) Both versions read the same unindexed GIST1M snapshot (1M rows × 960 dimensions, 12 splits). The table shows medians for query 0 over three runs after one warmup per version, comparing master `d90a902d` with the patch. | Executor | Storage | Peak scoring-process RSS (MiB), before → after | Query time (s), before → after | | --- | --- | ---: | ---: | | Local (2 CPUs) | Local Parquet | 4,865 → 795 (-83.7%) | 12.90 → 8.57 (-33.6%) | | Local (2 CPUs) | OSS | 5,158 → 683 (-86.8%) | 13.25 → 9.47 (-28.5%) | | Ray (2 workers) | Worker-local Parquet | 268 → 268 (0.0%) | 11.69 → 11.54 (-1.3%) | | Ray (2 workers) | OSS | 282 → 279 (-1.1%) | 18.35 → 18.75 (+2.2%) | Ray workers already score one split at a time from a batch reader, and their peak RSS stayed close across versions. The Ray rows report the largest scoring worker's peak and include the time to fetch the ten hits. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
