Abhinav Dixit created KAFKA-18018:
-------------------------------------
Summary: Consumption failing with DefaultStatePersister for 1
million records
Key: KAFKA-18018
URL: https://issues.apache.org/jira/browse/KAFKA-18018
Project: Kafka
Issue Type: Sub-task
Reporter: Abhinav Dixit
I was running some performance tests and I couldn't consume 1,000,000 records
using 10 share consumers. I've narrowed down the issue to
{{DefaultStatePersister}} . When I run the same tests using
{{{}NoOpShareStatePersister{}}}, then I don't see the reliability issue.
Steps to reproduce (could be maybe reproduced by other ways as well, but this
is how I noticed it consistently) - # Create a topic with a single partition
# Produce 1000 records into the topic and consume it with the help of 10 share
consumers.
# Again produce 1000 records into the topic and consume it with the help of 10
share consumers.
# Produce 1,000,000 records into the topic and consume it with the help of 10
share consumers.
You'll see the records not getting consumed even within 5 minutes (It shouldn't
take more than 5-6 seconds to consume 1,000,000 records with 10 share
consumers, based on my past experience)
PS - I haven't noticed any issues using console share consumers, so probably an
issue with scale
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
