I am currently using the Elasticsearch (ELK stack) and Redis is the current choice as broker.
I want to move to a distributed broker to make that layer more HA. Currently exploring kafka as a replacement. I have a few questions: 1. I read that kafka is designed to write contents to disk and this cannot be turned off. If everything is working properly on the elasticsearch side, the logs should be pulled off right away. Is there a setting i can use to hold the logs in page cache before writing to disk? 2. Does kafka work well in virtualized vmware environment? Does anyone has specs to be used for sustained 80k messages per second. I am thinking of using three kafka nodes to begin with. Sorry for the questions. I cannot find a really good book right now. -- Allen Michael Chan
