What mode are those sockets in (established, closed, etc)? Also, from the ip, could you tell whether those sockets are from the client or from the replica fetcher in the brokers.
Thanks, Jun On Tue, Jan 21, 2014 at 3:29 AM, Ahmy Yulrizka <a...@yulrizka.com> wrote: > We are running 3 kafka nodes, which servers 4 partition. > We have been experiencing weird behavior during network outage. > > we had been experiencing twice in the last couple of days. the previous one > took down all of the cluster. > while this one only 2 out of 3 survive. and 1 node became the leader of all > partition, and other node only in ISR of 1 partition (out of 4) > > my best guess now is that when the network down, the broker can't connect > to other broker to do replication and keep opening the socket > without closing it. But I'm not entirely sure about this. > > Is there any way to mitigate the problem ? or is there any configuration > options to stop this from happening again ? > > > The java/kafka process open too many socket file descriptor. > running `lsof -a -p 11818` yield thousand of this line. > > ... > java 11818 kafka 3059u sock 0,7 0t0 615637305 > can't identify protocol > java 11818 kafka 3060u sock 0,7 0t0 615637306 > can't identify protocol > java 11818 kafka 3061u sock 0,7 0t0 615637307 > can't identify protocol > java 11818 kafka 3062u sock 0,7 0t0 615637308 > can't identify protocol > java 11818 kafka 3063u sock 0,7 0t0 615637309 > can't identify protocol > java 11818 kafka 3064u sock 0,7 0t0 615637310 > can't identify protocol > java 11818 kafka 3065u sock 0,7 0t0 615637311 > can't identify protocol > ... > > i verify that the the open socket did not close when i repeated the command > after 2 minutes. > > > and the kafka log on the broken node, generate lots of error like this: > > [2014-01-21 04:21:48,819] 64573925 [kafka-acceptor] ERROR > kafka.network.Acceptor - Error in acceptor > java.io.IOException: Too many open files > at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method) > at > sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:165) > at kafka.network.Acceptor.accept(SocketServer.scala:200) > at kafka.network.Acceptor.run(SocketServer.scala:154) > at java.lang.Thread.run(Thread.java:701) > [2014-01-21 04:21:48,819] 64573925 [kafka-acceptor] ERROR > kafka.network.Acceptor - Error in acceptor > java.io.IOException: Too many open files > at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method) > at > sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:165) > at kafka.network.Acceptor.accept(SocketServer.scala:200) > at kafka.network.Acceptor.run(SocketServer.scala:154) > at java.lang.Thread.run(Thread.java:701) > [2014-01-21 04:21:48,811] 64573917 [ReplicaFetcherThread-0-1] INFO > kafka.consumer.SimpleConsumer - Reconnect due to socket error: null > [2014-01-21 04:21:48,819] 64573925 [ReplicaFetcherThread-0-1] WARN > kafka.server.ReplicaFetcherThread - [ReplicaFetcherThread-0-1], Error in > fetch Name: FetchRequest; Version: 0; CorrelationId: 74930218; ClientId: > ReplicaFetcherThread-0-1; ReplicaId: 2; MaxWait: 500 ms; MinBytes: 1 bytes; > RequestInfo: [some-topic,0] -> > PartitionFetchInfo(959825,1048576),[some-topic,3] -> > PartitionFetchInfo(551546,1048576) > java.net.SocketException: Too many open files > at sun.nio.ch.Net.socket0(Native Method) > at sun.nio.ch.Net.socket(Net.java:156) > at sun.nio.ch.SocketChannelImpl.<init>(SocketChannelImpl.java:102) > at > > sun.nio.ch.SelectorProviderImpl.openSocketChannel(SelectorProviderImpl.java:55) > at java.nio.channels.SocketChannel.open(SocketChannel.java:122) > at kafka.network.BlockingChannel.connect(BlockingChannel.scala:48) > at kafka.consumer.SimpleConsumer.connect(SimpleConsumer.scala:44) > at kafka.consumer.SimpleConsumer.reconnect(SimpleConsumer.scala:57) > at > kafka.consumer.SimpleConsumer.liftedTree1$1(SimpleConsumer.scala:79) > at > > kafka.consumer.SimpleConsumer.kafka$consumer$SimpleConsumer$$sendRequest(SimpleConsumer.scala:71) > at > > kafka.consumer.SimpleConsumer$$anonfun$fetch$1$$anonfun$apply$mcV$sp$1.apply$mcV$sp(SimpleConsumer.scala:110) > at > > kafka.consumer.SimpleConsumer$$anonfun$fetch$1$$anonfun$apply$mcV$sp$1.apply(SimpleConsumer.scala:110) > at > > kafka.consumer.SimpleConsumer$$anonfun$fetch$1$$anonfun$apply$mcV$sp$1.apply(SimpleConsumer.scala:110) > at kafka.metrics.KafkaTimer.time(KafkaTimer.scala:33) > at > > kafka.consumer.SimpleConsumer$$anonfun$fetch$1.apply$mcV$sp(SimpleConsumer.scala:109) > at > > kafka.consumer.SimpleConsumer$$anonfun$fetch$1.apply(SimpleConsumer.scala:109) > at > > kafka.consumer.SimpleConsumer$$anonfun$fetch$1.apply(SimpleConsumer.scala:109) > at kafka.metrics.KafkaTimer.time(KafkaTimer.scala:33) > at kafka.consumer.SimpleConsumer.fetch(SimpleConsumer.scala:108) > at > > kafka.server.AbstractFetcherThread.processFetchRequest(AbstractFetcherThread.scala:94) > at > kafka.server.AbstractFetcherThread.doWork(AbstractFetcherThread.scala:86) > at kafka.utils.ShutdownableThread.run(ShutdownableThread.scala:51) > > > -- > Ahmy Yulrizka > http://ahmy.yulrizka.com > @yulrizka >
