jimczi commented on PR #16684: URL: https://github.com/apache/lucene/pull/16684#issuecomment-5832949557
Thanks for digging into it, we ended up at the same conclusion on our side. On the file handles you raised, I tried to mitigate it by closing earlier. `StoredFieldsReader` now gets the `finishMerge` hook that `KnnVectorsReader` already has, called from `SegmentMerger.cleanupMerge`, so the extra open lives for the merge instead of for as long as the segment stays open. On NIOFS the handle comes back as soon as the merge ends. On mmap the unmap still waits for the segment's shared arena, since `MMapDirectory` groups arenas per segment, but that costs address space rather than a handle. On reusing one fd and mapping it twice: `MMapDirectory` closes the channel right after mapping, and two mappings from the same fd would also share `file_ra_state`, so the merge's readahead window and the `mmap_miss` counter would mix with the searchers'. A separate open avoids that. I will add a note to #13920 with what we found about read advice when the same file is mapped more than once. And if you are happy with a separate open for merging, I will do the same for vectors, where `getMergeInstance` re-advises the searchers' mapping today. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
