michaeljmarshall commented on PR #16145: URL: https://github.com/apache/lucene/pull/16145#issuecomment-6138876250
> Sample by page hash. It needs no state and no randomness, and the result is the same in every run I considered a position based hash, but I preferred the counter/seed solution. The `hash(offset)` solution is deterministic for a given offset and relies on variation of the offset to probe but still only ever probes for the same offsets, while the counter ensures probes every `SKIP`th call independently of the parameters. In my mind, the goal of the `isLoaded` probe is to give random feedback on a given cadence to determine whether the file is still in the page cache. If the probe is metadata dependent, then it seems it eventually ensures that pages for `hash(offset)` are loaded but not others. Call count based probing fails when there are cycles that line up with `SKIP`, so it has its own limitations, but I naively assert that the counter solution is less correlated than the hash solution, making it preferable. Please correct me if you see it differently though. > If every file starts confident, the slowdown goes away. What are the use cases where a file starts warm in the page cache? In my mind, these are the ones just written by flush or by merge. In the flush case, those files are likely to be small, and perhaps we could have a branched behavior for small files? In those cases, it doesn't make sense to perform 1024 calls to `isLoaded` anyway, and it might be significantly cheaper to execute the `isLoaded` call over the whole file. I am a new contributor though, so I don't know all of the nuances here and am open to guidance. I did have an agent look into whether it'd be easy to determine if a file was just written, but it didn't find anything satisfactory in the current feedback loops. As I write this though, I wonder if perhaps there is a cheap concurrent bounded queue implementation that could offer a hint on the right path to take? That implementation could serve as an at most once delivery of recently written files. Alternatively, we could consider a middle solution where we probe for fewer than 1024 calls but more than 3 calls (the current default). 1024 comes from the cost of a major page fault vs an unnecessary `isLoaded` call. Also, what is the methodology you're using that justifies the start confident conclusion? When is an `MemorySegmentIndexInput` opened? Is there generally an expectation that the underlying file is already hot? I will look into these soon, but posting in this comment now in case they're easy to answer. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
