I would encourage you todo so.
I also think its not reasonable behavior
On 13.02.2018 11:28, Wouter Bancken wrote:
We have upgraded our Kafka version as an attempt to solve this issue.
However, the issue is still present in Kafka 1.0.0.
Can I log a bug for this in JIRA?
On 5 February 2018 at 09:22, Wouter Bancken <wouter.banc...@aca-it.be>
The consumers in consumer group 'X' do not have a regex subscription
matching the newly created topic 'C'. They simply subscribe with
the subscribe(java.util.Collection<java.lang.String> topics) method on
topics 'A' and 'B'.
Shouldn't the consumer group have a different state from "Stable" during a
rebalancing regardless of the cause? How else can we determine the consumer
lag of the group during the rebalancing?
Have a look at our brand NEW job website: jobs.aca-it.be !
*ACA IT-Solutions NV*
*HQ:* Herkenrodesingel 8B 2.01 | 3500 Hasselt
T +32(0)11 26 50 10 | F +32(0)11 26 50 11
www.aca-it.be | Twitter <https://twitter.com/dorien_jorissen> | Facebook
On 5 February 2018 at 00:13, Hans Jespersen <h...@confluent.io> wrote:
Do the consumers in consumer group ‘X’ have a regex subscription that
matches the newly created topic ‘C’?
If they do then they will only discover this new topic once their ‘
metadata.max.age.ms’ metadata refresh interval has passed, which
defaults to 5 minutes.
metadata.max.age.ms The period of time in milliseconds after which
we force a refresh of metadata even if we haven't seen any partition
leadership changes to proactively discover any new brokers or partitions
On Feb 4, 2018, at 2:16 PM, Wouter Bancken <wouter.banc...@aca-it.be>
Thanks for the response!
However, I get this result for all topics, not just for the newly
1. I have a consumer group 'X' subscribed to topics 'A' and 'B' with
partition assignments and lag information. Consumer group 'X' is
2a. Topic 'C' is (being) created.
2b. During this creation, I do not have a partition assignment for
group 'X' for topics 'A' and 'B' but the consumer group is still
3. A second later: I have a partition assignment for consumer group 'X'
topics 'A' and 'B' again and the consumer group is still "Stable".
I expected the state of consumer group 'X' during step 2b to be
"PreparingRebalance" or "AwaitingSync".
On 4 February 2018 at 21:25, Hans Jespersen <h...@confluent.io> wrote:
I believe this is expected behavior.
If there are no subscriptions to a new topic, and therefor no partition
assignments, and definitely no committed offsets, then lag is an
concept. When the consumers subscribe to this new topic they may chose
start at the beginning or end of the commit log so the lag cannot be
predicted in advance.
On Feb 4, 2018, at 11:51 AM, Wouter Bancken <wouter.banc...@aca-it.be
Can anyone clarify if this is a bug in Kafka or the expected behavior?
On 30 January 2018 at 21:04, Wouter Bancken <wouter.banc...@aca-it.be
I'm trying to write an external tool to monitor consumer lag on
For this purpose, I'm using the kafka-consumer-groups tool to fetch
When using this tool, partition assignments seem to be unavailable
temporarily during the creation of a new topic even if the consumer
has no subscription on this new topic. This seems to match the
saying *"Topic metadata changes which have no impact on subscriptions
However, when this occurs I'd expect the state of the consumer to be
"PreparingRebalance" or "AwaitingSync" but it is simply "Stable".
Is this a bug in the tooling or is there a different way to obtain
correct offsets for a consumer group during a rebalance?
I'm using Kafka 10.2.1 but I haven't found any related issues in