jimczi commented on PR #16145:
URL: https://github.com/apache/lucene/pull/16145#issuecomment-6803316519

   +1 to merging this, bounding the blind spot is a clear improvement over main.
   
   For the follow-ups though, I'd stop trying to make the default right 
everywhere. Whether prefetch pays off depends on the file and on the 
deployment: rescoring vectors and stored fields are usually cold and read in 
batches, the terms dictionary and postings are usually in memory, unless the 
application decided its postings mostly live on disk. The page cache only tells 
us after the fact, so each new signal (getrusage, perf_event_open, more tuned 
constants) adds complexity and still won't be right for everyone.
   
   A simple default that the application can override seems far more 
achievable. We already do this for read advice (`setReadAdvice`) and preload 
(`setPreload`), from the file name and the format's hints. Prefetch could 
follow the same pattern: a policy on `MMapDirectory` with a default driven by 
the hints, where files with `NoReuseHint` always issue the madvise and 
everything else uses the backoff in this PR. Lucene keeps the mechanism simple, 
and the application, which knows its data, makes the call.
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]


---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to