michaeljmarshall commented on PR #16145: URL: https://github.com/apache/lucene/pull/16145#issuecomment-6475081922
@pauldoo - fair points. I'm thinking about this in terms of tail latency and the fact that these probes are per file so each file must discover the change in memory pressure independently. Given the challenge in finding the right values, maybe it'd be worth allowing systems to configure `SKIP` and `N` themselves. We'd still need defaults, but they wouldn't be as consequential. I'm also a relatively new contributor to Lucene. The great thing about the apache way is that we're all peers. :) @mikemccand - completely agree. The problem here is a lack of granular enough feedback loops. Naively, I wish `madvise` could return a hint about the state of the page cache, allowing us to skip the `mincore` call altogether. There's also a subtle problem in that we don't prefetch before all reads (we don't always know soon enough to issue the madvise call), so major page faults in themselves are not a fully accurate signal for whether we should issue the `madvise` call within the `prefetch()` call. However, as @pauldoo suggests, maybe tracking major page fault metrics at a system level could give us a sufficient hint to probe sooner? We could centralize the `perf_event_open` tracking (or possibly `getrusage`) and then prefetch callers could probe based on those metrics. A system with a lot of merge and indexing activity likely probes more, but perhaps it is correct enough. Finally, maybe the mistake is trying to solve such a general problem without input from the parent system. Perhaps there is an interface that could allow for more granular control per file type, and such an interface would allow those parent systems to implement alternative mechanisms like tracking `perf_event_open` metrics. (Though to be fair, users can already implement the `IndexInput` interface...) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
