jimczi commented on PR #16145: URL: https://github.com/apache/lucene/pull/16145#issuecomment-6803316519
+1 to merging this, bounding the blind spot is a clear improvement over main. For the follow-ups though, I'd stop trying to make the default right everywhere. Whether prefetch pays off depends on the file and on the deployment: rescoring vectors and stored fields are usually cold and read in batches, the terms dictionary and postings are usually in memory, unless the application decided its postings mostly live on disk. The page cache only tells us after the fact, so each new signal (getrusage, perf_event_open, more tuned constants) adds complexity and still won't be right for everyone. A simple default that the application can override seems far more achievable. We already do this for read advice (`setReadAdvice`) and preload (`setPreload`), from the file name and the format's hints. Prefetch could follow the same pattern: a policy on `MMapDirectory` with a default driven by the hints, where files with `NoReuseHint` always issue the madvise and everything else uses the backoff in this PR. Lucene keeps the mechanism simple, and the application, which knows its data, makes the call. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
