I've been having some issues with the New Consumer. I'm aware there is a
bug that has been fixed for 0.9.0.1, but is this the same thing?
I'm using manual partition assignment due to latency issues making it near
impossible to work with the group management features.
So, my consumer was going along fine for most of the day - it just consumes
from a topic with a single partition. However it has just stopped receiving
messages and I can see there is a backlog of around 100k messages to get
through. Since message consumption has stopped i get the below "Marking the
coordinator dead" log messages every 9 minutes. I have done multiple stack
dumps to see what is happening, one of which is below, and it is always
appears to be in the consumer.poll
So.. same bug as the one i believe is fixed on 0.9.0.1? In which case i'll
upgrade my client to the latest from the branch. Or is this something
different?
Thanks,
Damian
2016/02/13 00:07:57 131.73 MB/1.8 GB INFO
org.apache.kafka.clients.consumer.internals.AbstractCoordinator -
Marking the coordinator 2147479630 dead.
2016/02/13 00:16:57 151.75 MB/1.79 GB INFO
org.apache.kafka.clients.consumer.internals.AbstractCoordinator -
Marking the coordinator 2147479630 dead.
2016/02/13 00:25:57 181.07 MB/1.76 GB INFO
org.apache.kafka.clients.consumer.internals.AbstractCoordinator -
Marking the coordinator 2147479630 dead.
org.apache.kafka.clients.consumer.internals.AbstractCoordinator -
Marking the coordinator 2147479630 dead.
"poll-kafka-1" #45 prio=5 os_prio=0 tid=0x00007f7dba9da800 nid=0x52fd
runnable [0x00007f7cecbe3000]
java.lang.Thread.State: RUNNABLE
at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79)
at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
- locked <0x00000000ac6df4c8> (a sun.nio.ch.Util$2)
- locked <0x00000000ac6df4b0> (a
java.util.Collections$UnmodifiableSet)
- locked <0x00000000ac53db20> (a sun.nio.ch.EPollSelectorImpl)
at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
at
org.apache.kafka.common.network.Selector.select(Selector.java:425)
at org.apache.kafka.common.network.Selector.poll(Selector.java:254)
at
org.apache.kafka.clients.NetworkClient.poll(NetworkClient.java:270)
at
org.apache.kafka.clients.consumer.internals.ConsumerNetworkClient.clientPoll(ConsumerNetworkClient.java:303)
at
org.apache.kafka.clients.consumer.internals.ConsumerNetworkClient.poll(ConsumerNetworkClient.java:197)
at
org.apache.kafka.clients.consumer.internals.ConsumerNetworkClient.poll(ConsumerNetworkClient.java:187)
at
org.apache.kafka.clients.consumer.KafkaConsumer.pollOnce(KafkaConsumer.java:877)
at
org.apache.kafka.clients.consumer.KafkaConsumer.poll(KafkaConsumer.java:829)
