jasperjiaguo opened a new issue, #12166: URL: https://github.com/apache/pinot/issues/12166
Recently we have discovered the Pinot query latency can be impact by value of Linux's `read_ahead_kb`. Specifically we see a very high page fault count and severe tail latency shootup when the `read_ahead_kb` is set to a larger value in certain Linux distributions. As `read_ahead_kb` controls the read ahead during the access of mmap files, we think using larger value harms the queries having more random data access patterns. Theoretically it might benefit the opposite pattern but we have yet to see such a case. I think there are a few things that might worth doing: - In the short term we add this as a tip for Pinot admin in the OSS doc, so that it becomes a public knowledge - In the long term we may explore controlling this programmatically (like madvise in c), but it might be harder to do in Java - Revisit the mmap based segment cache Similar issues/analysis: https://smalldatum.blogspot.com/2014/05/the-impact-of-read-ahead-and-read-size.html https://github.com/elastic/elasticsearch/issues/27748 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
