Abuli Palagashvili created KAFKA-10480:
------------------------------------------
Summary: Kafka streams application is being stuck
Key: KAFKA-10480
URL: https://issues.apache.org/jira/browse/KAFKA-10480
Project: Kafka
Issue Type: Bug
Components: consumer, streams
Affects Versions: 2.2.0, 0.10.2.1
Reporter: Abuli Palagashvili
*prerequisites:*
* Kafka cluster running on version 0.10.2.1
* Topic with 24 partitions, load up to 20k RPS, stored bare Strings with null
keys
* Kafka-streams application that reads records from source topic and writes to
another, target partition defined from key extracted from record.Uses library
version 2.2.0
*Problem:*
After application start all goes ok, but sometimes I get this message:
2020-09-10 20:09:41 WARN AbstractCoordinator:1119 - [Consumer
clientId=sharder-application-1-8545e058-3494-4951-93d3-94bb4833be44-StreamThread-5-consumer,
groupId=sharder-application-1] This member will leave the group because
consumer poll timeout has expired. This means the time between subsequent calls
to poll() was longer than the configured max.poll.interval.ms, which typically
implies that the poll loop is spending too much time processing messages. You
can address this either by increasing max.poll.interval.ms or by reducing the
maximum size of batches returned in poll() with max.poll.records.
I tried playing with these config properties, but still face that problem.
My application processes records pretty fast and I think that it just loses
connections to kafka cluster.Another problem is that I can't handle group
member leaving, because application doesn't throw any exception and doesn't
change it's state so I can't catch that situation as one does it here.
https://dzone.com/articles/whats-the-proper-kubernetes-health-check-for-a-kaf
Maybe somebody also faced this?
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
