lizhanhui opened a new issue, #4465: URL: https://github.com/apache/rocketmq/issues/4465
RocketMQ is designed to be **write friendly**, using commit log file sequence to store all topics served by the broker. This works well since the majority of the read are performed when data are hot, thus no random read is not involved. Unfortunately, it is not always the case as more workloads are served. If a few subscribers read data that were kicked out of page-cache, severe read amplification and page-cache pollution are observed, bringing substantial impact to IO bandwidth, resulting in broker busy from time to time. Page-cache pollution, at present, cannot be fixed unless we turn to Direct IO and manage cache manually, now that flag POSIX_FADV_NOREUSE of posix_madvise in Linux is **[no-op](https://github.com/torvalds/linux/blob/master/mm/fadvise.c#L109)**, which is in sync with [its doc](https://linux.die.net/man/2/posix_fadvise). Read amplification may be mitigated through posix_madvise, yet by way of JNI. Along the way to identifying the performance issue, I found Lucene/elastic-search developers came across this issue years ago https://github.com/elastic/elasticsearch/issues/27748 https://blog.mikemccandless.com/2010/06/lucene-and-fadvisemadvise.html -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
