jimczi commented on PR #16684: URL: https://github.com/apache/lucene/pull/16684#issuecomment-5831413298
Thanks for looking. One correction on the mechanism first, since it's easy to check: read advice lives on the mapping, not on the file or the pages. `MADV_RANDOM` and `MADV_SEQUENTIAL` set `VM_RAND_READ` / `VM_SEQ_READ` on the VMAs of the range you pass. So a second open has its own advice, the mapping searches are on is untouched, and when the merge mapping is unmapped its flags go with it. The page cache is shared, which is the part we want. On opening a clone and overriding its advice: a clone shares the same `MemorySegment` array (`buildSlice` returns `segments` when it is a clone) and `updateReadAdvice` walks that array, so advising a clone advises the original. Only a fresh open has advice of its own. Doing this with `updateIOContext` is easy, nothing blocks it. My objection is not cost, it is that it makes the searchers' mapping sequential for as long as the merge runs. These are the big files, a `.fdt` is often larger than the whole page cache, which is why it is advised random in the first place, and the file a merge reads is a source that gets deleted at commit, so warming it cannot help a later search. That is what you said in #13920 as well, that signalling we read it once means it is good to drop it from the cache soon. The other thing the second open buys is that the directory gets to decide. `updateIOContext` is an mmap notion, it re-advises a mapping that already exists, so `DirectIODirectory` or a hybrid directory never sees the merge and cannot pick direct I/O or a different input for it. Re-opening is the only point where that choice can be made. The cost here is one extra mapping per stored fields reader that gets merged, opened lazily, page cache shared so nothing is cached twice, closed when the segment's reader is closed. If you would rather it went away as soon as the merge ends, we can add a finishMerge hook for stored fields like `KnnVectorsReader` has. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
