Hi everyone, I'm currently experiencing a situation that looks very similar to this[1]. I'm running Kafka 0.8.2.1 and we use it with Storm but don't use Storm's kafka spout since it doesn't quite work for our needs. I've been able to reproduce the issue locally by creating a storm topology with 1 worker (jvm) which has 8 spouts (threads) and each spout has its own ConsumerConnector and stream. The issue occurs when downstream processing continuously fails in which our spouts shut down their connector, waits 5s and then creates a new connector in order to replay the messages. After a flurry of error messages usually (syncedRebalance exceptions) everything settles down and our spouts continue to try and read new messages from the stream but nothing comes through. Looking at the thread dump shows all consumer fetcher threads parked unable to add more to their queue which could mean the queues are full,
java.lang.Thread.State: WAITING (parking) at sun.misc.Unsafe.park(Native Method) - parking to wait for <0x00000007bcc62e88> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject) at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) at java.util.concurrent.LinkedBlockingQueue.put(LinkedBlockingQueue.java:350) at kafka.consumer.PartitionTopicInfo.enqueue(PartitionTopicInfo.scala:60) at kafka.consumer.ConsumerFetcherThread.processPartitionData(ConsumerFetcherThread.scala:49) I realize this setup isn't the best and I would love to move to storm's kafka spout which looks like it avoids some of these problems but that may be a ways off. Any ideas or things to try? [1] - https://www.google.com/url?sa=t&rct=j&q=&esrc=s&source=web&cd=1&ved=0CB0QFjAAahUKEwjL5Z7T9bXIAhUY7WMKHQ35BJU&url=http%3A%2F%2Fmail-archives.apache.org%2Fmod_mbox%2Fkafka-users%2F201307.mbox%2F%253CCAFbh0Q3%3DugxWp-L%2BGaoycNbdW3WyCbNU4CrZ3qDWbP8uXFADRA%40mail.gmail.com%253E&usg=AFQjCNHF6myACUd50GrAqdgpwZflOsjYKg&sig2=61k2oqI9havLlhHuhdl3hw
