[jira] [Commented] (KAFKA-10513) Newly added topic or partitions are not assigned to running consumer groups using static membership
[
https://issues.apache.org/jira/browse/KAFKA-10513?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17204432#comment-17204432
]
Marlon Ou commented on KAFKA-10513:
---
Thank you both so much for the help! I tried setting metadata.max.age.ms to a
small value and it did work. The reason why I didn't see rebalance before is
that the default value for metadata.max.age.ms is 5 minutes and I never waited
past 5 mins for rebalancing.
> Newly added topic or partitions are not assigned to running consumer groups
> using static membership
> ---
>
> Key: KAFKA-10513
> URL: https://issues.apache.org/jira/browse/KAFKA-10513
> Project: Kafka
> Issue Type: Bug
> Components: consumer
>Affects Versions: 2.6.0
>Reporter: Marlon Ou
>Priority: Major
>
> If consumers are polling messages from a certain topic with static membership
> and we add new partitions to this topic while the consumers are running, no
> partition reassignment is ever triggered (and hence messages published into
> the new partitions are never consumed).
> To reproduce, simply set group instance IDs on the consumers:
> {code:java}
> props.setProperty("group.instance.id", instanceId);
> {code}
> And then while the static consumers are running, use Kafka's admin client to
> add more partitions to the topic:
> {code:java}
> adminClient.createPartitions(...)
> {code}
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (KAFKA-10513) Newly added topic or partitions are not assigned to running consumer groups using static membership
[
https://issues.apache.org/jira/browse/KAFKA-10513?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17203622#comment-17203622
]
Guozhang Wang commented on KAFKA-10513:
---
[~mou@ea] could you try setting the `metadata.max.age.ms` to a very small value
and retry this scenario again? [~bchen225242] and I looked at both client and
broker side code in 2.6 and we think it should still trigger rebalance.
> Newly added topic or partitions are not assigned to running consumer groups
> using static membership
> ---
>
> Key: KAFKA-10513
> URL: https://issues.apache.org/jira/browse/KAFKA-10513
> Project: Kafka
> Issue Type: Bug
> Components: consumer
>Affects Versions: 2.6.0
>Reporter: Marlon Ou
>Priority: Major
>
> If consumers are polling messages from a certain topic with static membership
> and we add new partitions to this topic while the consumers are running, no
> partition reassignment is ever triggered (and hence messages published into
> the new partitions are never consumed).
> To reproduce, simply set group instance IDs on the consumers:
> {code:java}
> props.setProperty("group.instance.id", instanceId);
> {code}
> And then while the static consumers are running, use Kafka's admin client to
> add more partitions to the topic:
> {code:java}
> adminClient.createPartitions(...)
> {code}
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (KAFKA-10513) Newly added topic or partitions are not assigned to running consumer groups using static membership
[
https://issues.apache.org/jira/browse/KAFKA-10513?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17203342#comment-17203342
]
Boyang Chen commented on KAFKA-10513:
-
My understanding is that the consumer was supposed to check the metadata for
topic partition, and rejoin as necessary when the partition number changes?
> Newly added topic or partitions are not assigned to running consumer groups
> using static membership
> ---
>
> Key: KAFKA-10513
> URL: https://issues.apache.org/jira/browse/KAFKA-10513
> Project: Kafka
> Issue Type: Bug
> Components: consumer
>Affects Versions: 2.6.0
>Reporter: Marlon Ou
>Priority: Major
>
> If consumers are polling messages from a certain topic with static membership
> and we add new partitions to this topic while the consumers are running, no
> partition reassignment is ever triggered (and hence messages published into
> the new partitions are never consumed).
> To reproduce, simply set group instance IDs on the consumers:
> {code:java}
> props.setProperty("group.instance.id", instanceId);
> {code}
> And then while the static consumers are running, use Kafka's admin client to
> add more partitions to the topic:
> {code:java}
> adminClient.createPartitions(...)
> {code}
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (KAFKA-10513) Newly added topic or partitions are not assigned to running consumer groups using static membership
[
https://issues.apache.org/jira/browse/KAFKA-10513?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17202991#comment-17202991
]
Guozhang Wang commented on KAFKA-10513:
---
This seems to be a valid issue after reading the source code, cc [~bchen225242]
to take another look.
> Newly added topic or partitions are not assigned to running consumer groups
> using static membership
> ---
>
> Key: KAFKA-10513
> URL: https://issues.apache.org/jira/browse/KAFKA-10513
> Project: Kafka
> Issue Type: Bug
> Components: consumer
>Affects Versions: 2.6.0
>Reporter: Marlon Ou
>Priority: Major
>
> If consumers are polling messages from a certain topic with static membership
> and we add new partitions to this topic while the consumers are running, no
> partition reassignment is ever triggered (and hence messages published into
> the new partitions are never consumed).
> To reproduce, simply set group instance IDs on the consumers:
> {code:java}
> props.setProperty("group.instance.id", instanceId);
> {code}
> And then while the static consumers are running, use Kafka's admin client to
> add more partitions to the topic:
> {code:java}
> adminClient.createPartitions(...)
> {code}
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
