You should check the kafka docs. You need to increase your open file descriptors.
Here's a link that describes how to do it. (Ignore that this is for hdp the ulimit part is what you need to pay attend to.) You can also check kafka docs too they say to change open file descriptors to 100,000 I believe. https://docs.hortonworks.com/HDPDocuments/HDP2/HDP-2.6.0/bk_security/content/kerb-config-limits.html On Jan 3, 2018 10:48 AM, "daniel brown" <daniel.w.bro...@gmail.com> wrote: Hello, Currently Zookeeper and kafka do not work appropriately and the logs are not streaming to logstash. The var-log partitions across the zookeeper cluster are filled up to 100 percent. We are getting the following messages when we investigated the logs and the services status: 1 . Zookeeper logs 2018-01-03 06:16:55,001 [myid:1] - WARN [SendWorker:2:QuorumCnxManager$SendWorker@736] - Send worker leaving thread 2018-01-03 06:23:08,943 [myid:1] - WARN [RecvWorker:3:QuorumCnxManager$RecvWorker@810] - Connection broken for id 3, my id = 1, error = java.io.EOFException at java.io.DataInputStream.readInt(DataInputStream.java:392) at org.apache.zookeeper.server.quorum.QuorumCnxManager$RecvWorker.run( QuorumCnxManager.java:795) 2018-01-03 06:23:08,943 [myid:1] - WARN [RecvWorker:3:QuorumCnxManager$RecvWorker@813] - Interrupting SendWorker 2018-01-03 06:23:08,943 [myid:1] - WARN [QuorumPeer[myid=1]/0.0.0.0:2181 :Follower@87] - Exception when following the leader java.io.EOFException at java.io.DataInputStream.readInt(DataInputStream.java:392) at org.apache. **Consumer check output kafka-console-consumer --bootstrap-server localhost:2181 --topic RCCSWA_DHCP_1 --from-beginning [2018-01-03 07:39:27,559] WARN Bootstrap broker localhost:2181 disconnected (org.apache.kafka.clients.NetworkClient) [2018-01-03 07:39:27,660] WARN Bootstrap broker localhost:2181 disconnected (org.apache.kafka.clients.NetworkClient) [2018-01-03 07:39:27,761] WARN Bootstrap broker localhost:2181 disconnected (org.apache.kafka.clients.NetworkClient) 2. Kafka Logs tail -f server.log.2018-01-02-13 at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:422) at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:250) at kafka.network.Acceptor.accept(SocketServer.scala:326) at kafka.network.Acceptor.run(SocketServer.scala:269) at java.lang.Thread.run(Thread.java:748) [2018-01-02 13:30:01,124] ERROR Error while accepting connection (kafka.network.Acceptor) java.io.IOException: Too many open files at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method) at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:422) at sun.nio.ch.ServerSocketChan Thanks.
