Well the validation is nice. Only seeing in this our test cluster (not production), but still no closer to a cause/resolution.
On Tue, Sep 11, 2018 at 4:04 PM Tyler Monahan <tjmonah...@gmail.com> wrote: > Matt, > > > I am seeing similar behavior with kafka 1.1.0 with 80 copies of the same > file being open. I am getting up to ~500,000 open files on each broker with > it dying around 600,000. I have yet to find any documentation saying how > many open files kafka should have per topic/partition. The only > documentation I have seen is having the limit set to at least 100,000 which > we are both well past. > > > [2018-09-07 14:35:33,872] ERROR Error while accepting connection > (kafka.network.Acceptor) > > java.io.IOException: Too many open files > > > Tyler Monahan > > > On 8/27/18, 12:29 PM, "Matt Kocubinski" <mkocubin...@gmail.com> wrote: > > > > A 5 node cluster is running Kafka 1.0.0 with 126 topics, most of them > with > > 24 partitions. Brokers are keeping around 800k file handles open > each, and > > regularly crashing with "Too many open file handles". > > > > Using one broker as a study I discovered There are ~13k log file > > segments/indices in the Kafka data dir discovered like: > > > > admin@kafka-1:/kafka/logs$ find . -type f | wc -l > > 12368 > > > > I've also noticed that a single segment file is open in 90 different > > threads: > > > > admin@kafka-1:/kafka/logs$ sudo lsof | grep > > "/site.time.tracking-6/00000000000000000297.log" | wc -l > > 90 > > > > This is about 1 per broker thread running! Extrapolating this nearly > adds > > up to 1.3mm, or the 800k I'm seeing. Something must be grossly > > mis-configured here. Why are the same segments being opened so many > times? >
