JIRA KAFKA-1716
While I don't have a patch to contribute today, per the Nag us note here: http://kafka.apache.org/contributing.html I'm wondering if we could get some attention to https://issues.apache.org/jira/browse/KAFKA-1716 We recently upgraded to Kafka 0.8.2.1 and hit a deadlock every time we shutdown our product. The stacktraces look very much like the ones in this JIRA. This has become a real problem for us. Thanks! Chris
[jira] [Commented] (KAFKA-1716) hang during shutdown of ZookeeperConsumerConnector
[
https://issues.apache.org/jira/browse/KAFKA-1716?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14652197#comment-14652197
]
Chris Barlock commented on KAFKA-1716:
--
We just upgraded to Kafka 0.8.2.1 and are now hitting this issue every single
time we shut down our environment. Is anyone looking at a fix?
hang during shutdown of ZookeeperConsumerConnector
--
Key: KAFKA-1716
URL: https://issues.apache.org/jira/browse/KAFKA-1716
Project: Kafka
Issue Type: Bug
Components: consumer
Affects Versions: 0.8.1.1
Reporter: Sean Fay
Assignee: Neha Narkhede
Attachments: after-shutdown.log, before-shutdown.log,
kafka-shutdown-stuck.log
It appears to be possible for {{ZookeeperConsumerConnector.shutdown()}} to
wedge in the case that some consumer fetcher threads receive messages during
the shutdown process.
Shutdown thread:
{code}-- Parking to wait for:
java/util/concurrent/CountDownLatch$Sync@0x2aaaf3ef06d0
at jrockit/vm/Locks.park0(J)V(Native Method)
at jrockit/vm/Locks.park(Locks.java:2230)
at sun/misc/Unsafe.park(ZJ)V(Native Method)
at java/util/concurrent/locks/LockSupport.park(LockSupport.java:156)
at
java/util/concurrent/locks/AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:811)
at
java/util/concurrent/locks/AbstractQueuedSynchronizer.doAcquireSharedInterruptibly(AbstractQueuedSynchronizer.java:969)
at
java/util/concurrent/locks/AbstractQueuedSynchronizer.acquireSharedInterruptibly(AbstractQueuedSynchronizer.java:1281)
at java/util/concurrent/CountDownLatch.await(CountDownLatch.java:207)
at kafka/utils/ShutdownableThread.shutdown(ShutdownableThread.scala:36)
at
kafka/server/AbstractFetcherThread.shutdown(AbstractFetcherThread.scala:71)
at
kafka/server/AbstractFetcherManager$$anonfun$closeAllFetchers$2.apply(AbstractFetcherManager.scala:121)
at
kafka/server/AbstractFetcherManager$$anonfun$closeAllFetchers$2.apply(AbstractFetcherManager.scala:120)
at
scala/collection/TraversableLike$WithFilter$$anonfun$foreach$1.apply(TraversableLike.scala:772)
at
scala/collection/mutable/HashMap$$anonfun$foreach$1.apply(HashMap.scala:98)
at
scala/collection/mutable/HashMap$$anonfun$foreach$1.apply(HashMap.scala:98)
at
scala/collection/mutable/HashTable$class.foreachEntry(HashTable.scala:226)
at scala/collection/mutable/HashMap.foreachEntry(HashMap.scala:39)
at scala/collection/mutable/HashMap.foreach(HashMap.scala:98)
at
scala/collection/TraversableLike$WithFilter.foreach(TraversableLike.scala:771)
at
kafka/server/AbstractFetcherManager.closeAllFetchers(AbstractFetcherManager.scala:120)
^-- Holding lock: java/lang/Object@0x2aaaebcc7318[thin lock]
at
kafka/consumer/ConsumerFetcherManager.stopConnections(ConsumerFetcherManager.scala:148)
at
kafka/consumer/ZookeeperConsumerConnector.liftedTree1$1(ZookeeperConsumerConnector.scala:171)
at
kafka/consumer/ZookeeperConsumerConnector.shutdown(ZookeeperConsumerConnector.scala:167){code}
ConsumerFetcherThread:
{code}-- Parking to wait for:
java/util/concurrent/locks/AbstractQueuedSynchronizer$ConditionObject@0x2aaaebcc7568
at jrockit/vm/Locks.park0(J)V(Native Method)
at jrockit/vm/Locks.park(Locks.java:2230)
at sun/misc/Unsafe.park(ZJ)V(Native Method)
at java/util/concurrent/locks/LockSupport.park(LockSupport.java:156)
at
java/util/concurrent/locks/AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
at
java/util/concurrent/LinkedBlockingQueue.put(LinkedBlockingQueue.java:306)
at kafka/consumer/PartitionTopicInfo.enqueue(PartitionTopicInfo.scala:60)
at
kafka/consumer/ConsumerFetcherThread.processPartitionData(ConsumerFetcherThread.scala:49)
at
kafka/server/AbstractFetcherThread$$anonfun$processFetchRequest$1$$anonfun$apply$mcV$sp$2.apply(AbstractFetcherThread.scala:130)
at
kafka/server/AbstractFetcherThread$$anonfun$processFetchRequest$1$$anonfun$apply$mcV$sp$2.apply(AbstractFetcherThread.scala:111)
at scala/collection/immutable/HashMap$HashMap1.foreach(HashMap.scala:224)
at
scala/collection/immutable/HashMap$HashTrieMap.foreach(HashMap.scala:403)
at
kafka/server/AbstractFetcherThread$$anonfun$processFetchRequest$1.apply$mcV$sp(AbstractFetcherThread.scala:111)
at
kafka/server/AbstractFetcherThread$$anonfun$processFetchRequest$1.apply(AbstractFetcherThread.scala:111)
at
kafka/server/AbstractFetcherThread$$anonfun$processFetchRequest$1.apply(AbstractFetcherThread.scala:111)
at kafka/utils/Utils$.inLock(Utils.scala:538)
at
kafka/server/AbstractFetcherThread.processFetchRequest(AbstractFetcherThread.scala:110
Re: Connection to zk shell on Kafka
I'm a user of Kafka/ZooKeeper not one of its developers, so I can't give you a technical explanation. I do agree that Kafka should ship the jline JAR if its zookeeper-shell depends on it. Chris From: Prabhjot Bharaj prabhbha...@gmail.com To: u...@zookeeper.apache.org, dev@kafka.apache.org Cc: us...@kafka.apache.org Date: 07/29/2015 01:27 PM Subject:Re: Connection to zk shell on Kafka Sure. It would be great if you could as well explain the reason why the absence of the jar creates this problem Also, I'm surprised that zookeeper that comes bundled with kafka 0.8.2 does not have the jline jar Regards, prabcs On Wed, Jul 29, 2015 at 10:45 PM, Chris Barlock barl...@us.ibm.com wrote: You need the jline JAR file that ships with ZooKeeper. Chris IBM Tivoli Systems Research Triangle Park, NC (919) 224-2240 Internet: barl...@us.ibm.com From: Prabhjot Bharaj prabhbha...@gmail.com To: us...@kafka.apache.org, u...@zookeeper.apache.org Date: 07/29/2015 01:13 PM Subject:Connection to zk shell on Kafka Hi folks, */kafka/bin# ./zookeeper-shell.sh localhost:2182/* *Connecting to localhost:2182/* *Welcome to ZooKeeper!* *JLine support is disabled* *WATCHER::* *WatchedEvent state:SyncConnected type:None path:null* *The shell never says connected* I'm running 5 node zookeeper cluster on 5-node kafka cluster (each kafka broker has 1 zookeeper server running) When I try connecting to the shell, the shell never says 'Connected' However, if I try connecting on another standalone zookeeper which has no links to kafka, I'm able to connect:- */kafka/bin# /zookeeper/scripts/zkCli.sh -server 127.0.0.1:2181 http://127.0.0.1:2181* *Connecting to 127.0.0.1:2181 http://127.0.0.1:2181* *Welcome to ZooKeeper!* *JLine support is enabled* *WATCHER::* *WatchedEvent state:SyncConnected type:None path:null* *[zk: 127.0.0.1:2181(CONNECTED) 0]* Am I missing something? Thanks, prabcs -- - There are only 10 types of people in the world: Those who understand binary, and those who don't
