Thanks Buvana. Is this happening in production only or can you also reproduce it in a test cluster? If the latter, would you be able to test the latest 0.10.0.0 release candidate? We fixed a few issues in the SocketServer.
Ismael On Tue, May 10, 2016 at 8:27 PM, Ramanan, Buvana (Nokia - US) < buvana.rama...@nokia.com> wrote: > Ismael, > > Created bug: > https://issues.apache.org/jira/browse/KAFKA-3689 > > Hope to get a quick resolution. > > Thanks, > Buvana > > -----Original Message----- > From: isma...@gmail.com [mailto:isma...@gmail.com] On Behalf Of EXT > Ismael Juma > Sent: Tuesday, May 10, 2016 10:54 AM > To: users@kafka.apache.org > Subject: Re: ERROR Processor got uncaught exception. > (kafka.network.Processor) > > OK, thanks. I suggest filing a bug in JIRA and please provide as much > information as possible (steps to reproduce would be ideal, but sometimes > that is hard to do). It does look like a Kafka bug. > > Ismael > > On Tue, May 10, 2016 at 2:45 PM, Ramanan, Buvana (Nokia - US) < > buvana.rama...@nokia.com> wrote: > > > Ismael, > > > > Version 0.9.0.1 > > > > Do you have any idea how to prevent this from happening? Is it a Kafka > > issue? > > > > -Buvana > > > > -----Original Message----- > > From: isma...@gmail.com [mailto:isma...@gmail.com] On Behalf Of EXT > > Ismael Juma > > Sent: Monday, May 09, 2016 8:02 PM > > To: users@kafka.apache.org > > Subject: Re: ERROR Processor got uncaught exception. > > (kafka.network.Processor) > > > > Hi Buvana, > > > > What is the version of the broker? It looks like the state of connection > > counts for quotas became inconsistent, that is, a given client ip address > > has no connections according to the internal state, but a disconnection > for > > that ip happened. > > > > Ismael > > > > On Tue, May 10, 2016 at 12:14 AM, Ramanan, Buvana (Nokia - US) < > > buvana.rama...@nokia.com> wrote: > > > > > Hello, > > > > > > We are trying to understand the following error in one of the brokers > in > > > our cluster. It's a 3 broker cluster (all 3 servers are identical - > > Intel > > > Xeon E5-2670 @2.6GHz, 8cores, 16 threads 64 GB RAM & 1 TB Disk), > serving > > > about 8000 topics, single partitioned & replication factor = 3. Each > > topic > > > gets data at a low rate - 200 bytes per sec. Cluster is managed by 3 > > > server ZK cluster (these servers are different from Kafka broker > > servers). > > > All 6 servers are connected via 10G switch. Producers run from external > > > servers. > > > > > > Leaders are balanced across the topics. > > > > > > This error message occurs repeatedly in one of the servers. Between the > > > hours of 10:30am and 1:30pm today, there were about 10 Million such > > > occurrences. > > > > > > We tuned Linux network params (somaxconn, rmem_max, wmem_max, etc). I > can > > > share the current settings if needed. > > > > > > Please let me know what is causing this error and a way to overcome it. > > > > > > Regards, > > > Buvana > > > > > > [2016-05-09 10:38:43,932] ERROR Processor got uncaught exception. > > > (kafka.network.Processor) > > > java.lang.IllegalArgumentException: Attempted to decrease connection > > count > > > for address with no connections, address: /X.Y.Z.144 (actual network > > > address masked) > > > at > > > kafka.network.ConnectionQuotas$$anonfun$9.apply(SocketServer.scala:565) > > > at > > > kafka.network.ConnectionQuotas$$anonfun$9.apply(SocketServer.scala:565) > > > at scala.collection.MapLike$class.getOrElse(MapLike.scala:128) > > > at scala.collection.AbstractMap.getOrElse(Map.scala:59) > > > at kafka.network.ConnectionQuotas.dec(SocketServer.scala:564) > > > at > > > kafka.network.Processor$$anonfun$run$13.apply(SocketServer.scala:450) > > > at > > > kafka.network.Processor$$anonfun$run$13.apply(SocketServer.scala:445) > > > at scala.collection.Iterator$class.foreach(Iterator.scala:742) > > > at > scala.collection.AbstractIterator.foreach(Iterator.scala:1194) > > > at > > > scala.collection.IterableLike$class.foreach(IterableLike.scala:72) > > > at scala.collection.AbstractIterable.foreach(Iterable.scala:54) > > > at kafka.network.Processor.run(SocketServer.scala:445) > > > at java.lang.Thread.run(Thread.java:745) > > > [2016-05-09 10:38:43,932] ERROR Processor got uncaught exception. > > > (kafka.network.Processor) > > > java.lang.IllegalArgumentException: Attempted to decrease connection > > count > > > for address with no connections, address: /X.Y.Z.144 > > > at > > > kafka.network.ConnectionQuotas$$anonfun$9.apply(SocketServer.scala:565) > > > at > > > kafka.network.ConnectionQuotas$$anonfun$9.apply(SocketServer.scala:565) > > > at scala.collection.MapLike$class.getOrElse(MapLike.scala:128) > > > at scala.collection.AbstractMap.getOrElse(Map.scala:59) > > > at kafka.network.ConnectionQuotas.dec(SocketServer.scala:564) > > > at > > > kafka.network.Processor$$anonfun$run$13.apply(SocketServer.scala:450) > > > at > > > kafka.network.Processor$$anonfun$run$13.apply(SocketServer.scala:445) > > > at scala.collection.Iterator$class.foreach(Iterator.scala:742) > > > at > scala.collection.AbstractIterator.foreach(Iterator.scala:1194) > > > at > > > scala.collection.IterableLike$class.foreach(IterableLike.scala:72) > > > at scala.collection.AbstractIterable.foreach(Iterable.scala:54) > > > at kafka.network.Processor.run(SocketServer.scala:445) > > > at java.lang.Thread.run(Thread.java:745) > > > > > >
