Hey Guozhang, Thanks a lot for these insights. We are facing the exact same problem as Tianji. Our commit frequency is also quite high. We flush almost around 16K messages per minute to Kafka at the end of the topology.
Another issue that we are facing is that rocksdb is not deleting old data. We have set the time window retention duration to 1 hour, but the disk size is constantly increasing. Ideally, the disk utilisation graph should plateau after some time. We would like to apply the config change suggestions that you have given. But we are on Kafka 0.10.0.1. And from the docs, it seems rocksdb.config.setter is not available for this version. Is there any other way for us to configure rocksdb? Does Kafka 0.10.0.1 emit any rocksdb related metrics that we can monitor and set up alerts on? Thanks!
