Hello,
Currently Zookeeper and kafka do not work appropriately and the logs are
not streaming to logstash. The var-log partitions across the zookeeper
cluster are filled up to 100 percent. We are getting the following
messages when we investigated the logs and the services status:
1 . Zookeeper logs
2018-01-03 06:16:55,001 [myid:1] - WARN
[SendWorker:2:QuorumCnxManager$SendWorker@736] - Send worker leaving thread
2018-01-03 06:23:08,943 [myid:1] - WARN
[RecvWorker:3:QuorumCnxManager$RecvWorker@810] - Connection broken for id
3, my id = 1, error = java.io.EOFException
at java.io.DataInputStream.readInt(DataInputStream.java:392)
at
org.apache.zookeeper.server.quorum.QuorumCnxManager$RecvWorker.run(QuorumCnxManager.java:795)
2018-01-03 06:23:08,943 [myid:1] - WARN
[RecvWorker:3:QuorumCnxManager$RecvWorker@813] - Interrupting SendWorker
2018-01-03 06:23:08,943 [myid:1] - WARN [QuorumPeer[myid=1]/0.0.0.0:2181
:Follower@87] - Exception when following the leader java.io.EOFException
at java.io.DataInputStream.readInt(DataInputStream.java:392)
at org.apache.
**Consumer check output
kafka-console-consumer --bootstrap-server localhost:2181 --topic
RCCSWA_DHCP_1 --from-beginning
[2018-01-03 07:39:27,559] WARN Bootstrap broker localhost:2181 disconnected
(org.apache.kafka.clients.NetworkClient)
[2018-01-03 07:39:27,660] WARN Bootstrap broker localhost:2181 disconnected
(org.apache.kafka.clients.NetworkClient)
[2018-01-03 07:39:27,761] WARN Bootstrap broker localhost:2181 disconnected
(org.apache.kafka.clients.NetworkClient)
2. Kafka Logs
tail -f server.log.2018-01-02-13
at
sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:422)
at
sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:250)
at kafka.network.Acceptor.accept(SocketServer.scala:326)
at kafka.network.Acceptor.run(SocketServer.scala:269)
at java.lang.Thread.run(Thread.java:748)
[2018-01-02 13:30:01,124] ERROR Error while accepting connection
(kafka.network.Acceptor)
java.io.IOException: Too many open files
at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)
at
sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:422)
at sun.nio.ch.ServerSocketChan
Thanks.
