Matt,
I am seeing similar behavior with kafka 1.1.0 with 80 copies of the same
file being open. I am getting up to ~500,000 open files on each broker with
it dying around 600,000. I have yet to find any documentation saying how
many open files kafka should have per topic/partition. The only
documentation I have seen is having the limit set to at least 100,000 which
we are both well past.
[2018-09-07 14:35:33,872] ERROR Error while accepting connection
(kafka.network.Acceptor)
java.io.IOException: Too many open files
Tyler Monahan
On 8/27/18, 12:29 PM, "Matt Kocubinski" <mkocubin...@gmail.com> wrote:
A 5 node cluster is running Kafka 1.0.0 with 126 topics, most of them
with
24 partitions. Brokers are keeping around 800k file handles open each,
and
regularly crashing with "Too many open file handles".
Using one broker as a study I discovered There are ~13k log file
segments/indices in the Kafka data dir discovered like:
admin@kafka-1:/kafka/logs$ find . -type f | wc -l
12368
I've also noticed that a single segment file is open in 90 different
threads:
admin@kafka-1:/kafka/logs$ sudo lsof | grep
"/site.time.tracking-6/00000000000000000297.log" | wc -l
90
This is about 1 per broker thread running! Extrapolating this nearly
adds
up to 1.3mm, or the 800k I'm seeing. Something must be grossly
mis-configured here. Why are the same segments being opened so many
times?
