JingsongLi opened a new pull request, #11289: URL: https://github.com/apache/paimon/pull/11289
### Purpose A Data Evolution vector batch read retains every split future and merges all candidate scores for every query. A query batch over ordinary range-based indexes can run out of driver heap even when the final top-K results are small. Bound the outstanding reads by the configured read parallelism. Consume completed splits in input order and keep a per-query heap sized to the search or refinement limit. Track duplicate row IDs in overlapping input ranges so the first indexed score still wins when another split or a raw search returns the same row. ### Tests - Ran `VectorSearchBuilderTest`, `VectorSearchRefineOptionsTest` and `ScoredGlobalIndexResultTest` on Java 8 (65 cases). - Added an incremental-consumption regression with overlapping indexed ranges and a raw result sharing an evicted row. - Replayed 256 queries at K=100 on 64 ordinary IVF-FLAT files (32,768 vectors, four reader threads). Master ran out of heap at 128 MiB. The change completed with 128 MiB and returned byte-identical row IDs and float score bits to master with 1 GiB across all 25,600 hits. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
