A 5 node cluster is running Kafka 1.0.0 with 126 topics, most of them with 24 partitions. Brokers are keeping around 800k file handles open each, and regularly crashing with "Too many open file handles".
Using one broker as a study I discovered There are ~13k log file segments/indices in the Kafka data dir discovered like: admin@kafka-1:/kafka/logs$ find . -type f | wc -l 12368 I've also noticed that a single segment file is open in 90 different threads: admin@kafka-1:/kafka/logs$ sudo lsof | grep "/site.time.tracking-6/00000000000000000297.log" | wc -l 90 This is about 1 per broker thread running! Extrapolating this nearly adds up to 1.3mm, or the 800k I'm seeing. Something must be grossly mis-configured here. Why are the same segments being opened so many times?
