Re: Kafka Consumer Offsets unavailable during rebalancing

I logged https://issues.apache.org/jira/browse/KAFKA-6557 Best regards, Wouter On 13 February 2018 at 12:50, Jan Filipiak wrote: > I would encourage you todo so. > I also think its not reasonable behavior > > On 13.02.2018 11:28, Wouter Bancken wrote: > >> We have

Re: Kafka Consumer Offsets unavailable during rebalancing

I would encourage you todo so. I also think its not reasonable behavior On 13.02.2018 11:28, Wouter Bancken wrote: We have upgraded our Kafka version as an attempt to solve this issue. However, the issue is still present in Kafka 1.0.0. Can I log a bug for this in JIRA? Wouter On 5 February

Re: Kafka Consumer Offsets unavailable during rebalancing

We have upgraded our Kafka version as an attempt to solve this issue. However, the issue is still present in Kafka 1.0.0. Can I log a bug for this in JIRA? Wouter On 5 February 2018 at 09:22, Wouter Bancken wrote: > The consumers in consumer group 'X' do not have a

Re: Kafka Consumer Offsets unavailable during rebalancing

The consumers in consumer group 'X' do not have a regex subscription matching the newly created topic 'C'. They simply subscribe with the subscribe(java.util.Collection topics) method on topics 'A' and 'B'. Shouldn't the consumer group have a different state from "Stable" during a rebalancing

Re: Kafka Consumer Offsets unavailable during rebalancing

Do the consumers in consumer group ‘X’ have a regex subscription that matches the newly created topic ‘C’? If they do then they will only discover this new topic once their ‘metadata.max.age.ms’ metadata refresh interval has passed, which defaults to 5 minutes. metadata.max.age.ms The

Re: Kafka Consumer Offsets unavailable during rebalancing

Hi Hans, Thanks for the response! However, I get this result for all topics, not just for the newly created topic. Situation sketch: 1. I have a consumer group 'X' subscribed to topics 'A' and 'B' with partition assignments and lag information. Consumer group 'X' is "Stable". 2a. Topic 'C' is

Re: Kafka Consumer Offsets unavailable during rebalancing

I believe this is expected behavior. If there are no subscriptions to a new topic, and therefor no partition assignments, and definitely no committed offsets, then lag is an undefined concept. When the consumers subscribe to this new topic they may chose to start at the beginning or end of the

Re: Kafka Consumer Offsets unavailable during rebalancing

Can anyone clarify if this is a bug in Kafka or the expected behavior? Best regards, Wouter On 30 January 2018 at 21:04, Wouter Bancken wrote: > Hi, > > I'm trying to write an external tool to monitor consumer lag on Apache > Kafka. > > For this purpose, I'm using

Kafka Consumer Offsets unavailable during rebalancing

Hi, I'm trying to write an external tool to monitor consumer lag on Apache Kafka. For this purpose, I'm using the kafka-consumer-groups tool to fetch the consumer offsets. When using this tool, partition assignments seem to be unavailable temporarily during the creation of a new topic even if