JunRuiLee commented on PR #556: URL: https://github.com/apache/paimon-rust/pull/556#issuecomment-5043541332
Thanks @JingsongLi, you're exactly right — the limit was applied independently at each level, so a single search could reach N×N concurrent exact-file reads (and concurrent queries multiplied it further). Fixed in 5293431. Instead of two independent `buffer_unordered(N)` limits, the search now threads a single shared `Arc<Semaphore>` of N permits through the whole search, mirroring Java's single shared `GlobalIndexReadThreadPool`. A permit is acquired only around leaf exact-file I/O; the per-bucket orchestration never holds one, so it can't starve leaf work — the async analogue of the "start from the caller" note in `PrimaryKeyVectorRead.searchBuckets`. Total in-flight exact-file searches are now capped at N overall rather than per bucket, and the `concurrency == 1` path stays strictly sequential. I also added `search_candidates_peak_concurrency_capped_across_buckets`, which reproduces your scenario (two buckets, two files each, `concurrency = 2`): it instruments the exact-file search to record the peak number of simultaneously active searches and asserts it stays ≤ N. I confirmed the test observes peak = 4 and fails without the shared budget, and peak = 2 (passing) with it. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
