[ 
https://issues.apache.org/jira/browse/KAFKA-4299?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Juho Autio updated KAFKA-4299:
------------------------------
    Description: 
I increased partitions for one existing topic (2->10), but was surprised to see 
that it entirely reset the committed offsets of my consumer group.

All topics & partitions were reset to the earliest offset available, and the 
consumer read everything again.

Documentation doesn't mention anything like this. Is this how it's supposed to 
work, or a bug?

I would've expected the consumer offsets to not decrease at all, especially for 
the topics that I didn't even touch.

For the altered topic I would've expected that consuming the previously 
existing partitions 0 and 1 would've continued from the position where they 
were, and naturally starting to read the new added partitions from 0.

I added partitions according to the "Modifying topics" section of Kafka 0.10.0 
Documentation:

{quote}
To add partitions you can do
{code}
 > bin/kafka-topics.sh --zookeeper $ZOOKEEPER_HOST --alter --topic 
 > altered_topic --partitions 10
{code}
{quote}

Previously this topic had 2 partitions.

For the consumer I'm using 
{{kafka.javaapi.consumer.ConsumerConnector.createMessageStreamsByFilter()}}.

And version is:

{code}
<groupId>org.apache.kafka</groupId>
<artifactId>kafka_2.11</artifactId>
<version>0.10.0.1</version>
{code}

Kafka cluster itself is {{kafka_2.11-0.10.0.1}}.

This is quite problematic because we can't afford waiting for consumers to read 
the full buffer from the beginning (for all topics!) when increasing partitions 
for a topic.


Some possibly relevant settings we have for the consumer:

{code}
kafka.partition.assignment.strategy = "range"
kafka.auto.offset.reset = "smallest"
kafka.auto.commit.enable = "false"
kafka.offsets.storage = "kafka"
kafka.dual.commit.enabled = false
kafka.consumer.timeout.ms = "2000"
kafka.auto.create.topics.enable = true
{code}

  was:
I increased partitions for one existing topic (2->10), but was surprised to see 
that it entirely reset the committed offsets of my consumer group.

All topics & partitions were reset to the earliest offset available, and the 
consumer read everything again.

Documentation doesn't mention anything like this. Is this how it's supposed to 
work, or a bug?

I would've expected the consumer offsets to not decrease at all, especially for 
the topics that I didn't even touch.

For the altered topic I would've expected that consuming the previously 
existing partitions 0 and 1 would've continued from the position where they 
were, and naturally starting to read the new added partitions from 0.

I added partitions according to the "Modifying topics" section of Kafka 0.10.0 
Documentation:

{quote}
To add partitions you can do
{code}
 > bin/kafka-topics.sh --zookeeper $ZOOKEEPER_HOST --alter --topic 
 > altered_topic --partitions 10
{code}
{quote}

Previously this topic had 2 partitions.

For the consumer I'm using 
{{kafka.javaapi.consumer.ConsumerConnector.createMessageStreamsByFilter()}}.

And version is:

{code}
<groupId>org.apache.kafka</groupId>
<artifactId>kafka_2.11</artifactId>
<version>0.10.0.1</version>
{code}

Kafka cluster itself is {{kafka_2.11-0.10.0.1}}.

This is quite problematic because we can't afford waiting for consumers to read 
the full buffer from the beginning (for all topics!) when increasing partitions 
for a topic.


> Consumer offsets reset for all topics after increasing partitions for one 
> topic
> -------------------------------------------------------------------------------
>
>                 Key: KAFKA-4299
>                 URL: https://issues.apache.org/jira/browse/KAFKA-4299
>             Project: Kafka
>          Issue Type: Bug
>    Affects Versions: 0.10.0.1
>            Reporter: Juho Autio
>
> I increased partitions for one existing topic (2->10), but was surprised to 
> see that it entirely reset the committed offsets of my consumer group.
> All topics & partitions were reset to the earliest offset available, and the 
> consumer read everything again.
> Documentation doesn't mention anything like this. Is this how it's supposed 
> to work, or a bug?
> I would've expected the consumer offsets to not decrease at all, especially 
> for the topics that I didn't even touch.
> For the altered topic I would've expected that consuming the previously 
> existing partitions 0 and 1 would've continued from the position where they 
> were, and naturally starting to read the new added partitions from 0.
> I added partitions according to the "Modifying topics" section of Kafka 
> 0.10.0 Documentation:
> {quote}
> To add partitions you can do
> {code}
>  > bin/kafka-topics.sh --zookeeper $ZOOKEEPER_HOST --alter --topic 
> altered_topic --partitions 10
> {code}
> {quote}
> Previously this topic had 2 partitions.
> For the consumer I'm using 
> {{kafka.javaapi.consumer.ConsumerConnector.createMessageStreamsByFilter()}}.
> And version is:
> {code}
> <groupId>org.apache.kafka</groupId>
> <artifactId>kafka_2.11</artifactId>
> <version>0.10.0.1</version>
> {code}
> Kafka cluster itself is {{kafka_2.11-0.10.0.1}}.
> This is quite problematic because we can't afford waiting for consumers to 
> read the full buffer from the beginning (for all topics!) when increasing 
> partitions for a topic.
> Some possibly relevant settings we have for the consumer:
> {code}
> kafka.partition.assignment.strategy = "range"
> kafka.auto.offset.reset = "smallest"
> kafka.auto.commit.enable = "false"
> kafka.offsets.storage = "kafka"
> kafka.dual.commit.enabled = false
> kafka.consumer.timeout.ms = "2000"
> kafka.auto.create.topics.enable = true
> {code}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Reply via email to