Prakash Patel created KAFKA-12821:
-------------------------------------
Summary: Kafka consumer skip reading from single partition
Key: KAFKA-12821
URL: https://issues.apache.org/jira/browse/KAFKA-12821
Project: Kafka
Issue Type: Bug
Components: clients
Affects Versions: 2.3.0
Reporter: Prakash Patel
I have logstash running on Kubernetes and consuming events from 100+
topics.Each topic have 3+ partitions. Thus Single consumer group subscribed to
100+ topics and consumes events from 500+ partitions. Below are the Kafka
consumers configuration.I am running 10 instances of logstash on kubernetes so
total 100 consumer threads consuming from 500 partitions.
{quote}auto_offset_reset => "earliest"
enable_auto_commit => "true"
group_id => "test_logstash"
consumer_threads => "10"
max_poll_records => "500"
heartbeat_interval_ms => "9000"
session_timeout_ms => "30000"
fetch_max_bytes => "10485760"
max_partition_fetch_bytes => "524288"
client_id => "test_logstash"
decorate_events => true
partition_assignment_strategy =>
"org.apache.kafka.clients.consumer.RoundRobinAssignor"
{quote}
Kafka consumer stops reading data from single partition although its subscribed
and assigned to consumer group.
There is no rebalance or any error or info message Kafka consumer throws on
application side.
Restating the consumer solve the problem. This is intermittent issue and it can
happen with any partition of any topic consumer is subscribed/assigned.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
