adriangb commented on PR #11459: URL: https://github.com/apache/arrow-rs/pull/11459#issuecomment-6093407030
Thanks, it was indeed slower. Two reasons: 1. The optimization this was targeting was not reflected in the benchmarks. I added a benchmark now (8087cef). 2. `release_ranges` was not the bottleneck. `clear_ranges` rebuilt all buffers for each row group, so the binary search in `release_ranges` alone gave only 3-7%. I added a commit that fixes `clear_ranges` the same way (04faa64). Where that leaves us: `prefetch` goes from 478 ms to 19.2 ms at 30,000 buffers and is now linear in the number of buffers. Of that, the `clear_ranges` fix gives about 6x and the `VecDeque` commit a further 3.8x. That said the `VecDequeue` change is a tradeoff: it's ~5% slower when we have few buffers but ~3x faster when we have many. I think it's worth keeping. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
