Juho Autio created KAFKA-4299:

             Summary: Consumer offsets reset for all topics after increasing 
partitions for one topic
                 Key: KAFKA-4299
                 URL: https://issues.apache.org/jira/browse/KAFKA-4299
             Project: Kafka
          Issue Type: Bug
    Affects Versions:
            Reporter: Juho Autio

I increased partitions for one existing topic (2->10), but was surprised to see 
that it entirely reset the committed offsets of my consumer group.

All topics & partitions were reset to the earliest offset available, and the 
consumer read everything again.

Documentation doesn't mention anything like this. Is this how it's supposed to 
work, or a bug?

I would've expected the consumer offsets to not decrease at all, especially for 
the topics that I didn't even touch.

For the altered topic I would've expected that consuming the previously 
existing partitions 0 and 1 would've continued from the position where they 
were, and naturally starting to read the new added partitions from 0.

I added partitions according to the "Modifying topics" section of Kafka 0.10.0 

To add partitions you can do
 > bin/kafka-topics.sh --zookeeper $ZOOKEEPER_HOST --alter --topic 
 > altered_topic --partitions 10

Previously this topic had 2 partitions.

For the consumer I'm using 

And version is:


Kafka cluster itself is {{kafka_2.11-}}.

This is quite problematic because we can't afford waiting for consumers to read 
the full buffer from the beginning (for all topics!) when increasing partitions 
for a topic.

This message was sent by Atlassian JIRA

Reply via email to