unsubscribe On Wed, Aug 21, 2019 at 2:23 AM sampath kumar <sampath...@gmail.com> wrote:
> Hi, > > Using Broker 5.3.0, new consumers(Consumers managed by brokers). Brokers > are deployed in a Kubernetes environment > > Number of brokers : 3, Number of 3 Zookeeper setup > > One of the Topic "inventory.request" we have 3 replication, with insync > replicas configured as 2 and partition count is 1024 > > We have 20 instances of microservice subscribe to the above topic, each > instance will have 48 consumers registered as a group "agent.group.inv" > > Issue : > > Here some times a couple of the consumers suddenly stopped receiving the > request, and lag seems to keep increasing. Only option to recover it > restart the consumers and invokes rebalancing > > ``` agent.group.inv inventory.request 543 17423 17612 > 189 agent19.inv.35-6e6eb252-8d26-489b-8d7f-53b25f182f30 > /10.200.187.103 agent19.inv.35 ``` > > we checked the thread dump of the consumer, the consumer keeps performing > polling and assigned with partitions, However not receiving the any > messages > > ``` "inventory.request-agent19.inv.35" #499 prio=1 os_prio=4 > tid=0x00007f88a855b000 nid=0x389 runnable [0x00007f87e8be6000] > java.lang.Thread.State: RUNNABLE > at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) > at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) > at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) > at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) > - locked <0x00000000aa502730> (a sun.nio.ch.Util$3) > - locked <0x00000000aa5026b0> (a > java.util.Collections$UnmodifiableSet) > - locked <0x00000000aa502668> (a sun.nio.ch.EPollSelectorImpl) > at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) > at > org.apache.kafka.common.network.Selector.select(Selector.java:794) > at org.apache.kafka.common.network.Selector.poll(Selector.java:467) > at > org.apache.kafka.clients.NetworkClient.poll(NetworkClient.java:539) > at > > org.apache.kafka.clients.consumer.internals.ConsumerNetworkClient.poll(ConsumerNetworkClient.java:262) > at > > org.apache.kafka.clients.consumer.internals.ConsumerNetworkClient.poll(ConsumerNetworkClient.java:233) > at > > org.apache.kafka.clients.consumer.KafkaConsumer.pollForFetches(KafkaConsumer.java:1281) > at > > org.apache.kafka.clients.consumer.KafkaConsumer.poll(KafkaConsumer.java:1225) > at > > org.apache.kafka.clients.consumer.KafkaConsumer.poll(KafkaConsumer.java:1159) > ``` > > > No errors are observed consumer client, brokers and also resource issue not > seen. > > Can you please help us in identifying the root cause for this consumer > client behavior? > > Please let me know if any other details required? > > -- > Regards, > Sampath >
