Re: Moving data from one cluster to another with Kafka Streams

> Kafka Streams only work with a single cluster. Ok, that’s what I was thinking after I looked at it more. > Thus, you would need to either transform the data first and replicate > the output topic to the target cluster, or replicate first and transform > within the target cluster. I don’t

Re: Kafka Consumer Offsets unavailable during rebalancing

Do the consumers in consumer group ‘X’ have a regex subscription that matches the newly created topic ‘C’? If they do then they will only discover this new topic once their ‘metadata.max.age.ms’ metadata refresh interval has passed, which defaults to 5 minutes. metadata.max.age.ms The

Re: Kafka Consumer Offsets unavailable during rebalancing

Hi Hans, Thanks for the response! However, I get this result for all topics, not just for the newly created topic. Situation sketch: 1. I have a consumer group 'X' subscribed to topics 'A' and 'B' with partition assignments and lag information. Consumer group 'X' is "Stable". 2a. Topic 'C' is

Re: Kafka Consumer Offsets unavailable during rebalancing

I believe this is expected behavior. If there are no subscriptions to a new topic, and therefor no partition assignments, and definitely no committed offsets, then lag is an undefined concept. When the consumers subscribe to this new topic they may chose to start at the beginning or end of the

Re: Kafka Consumer Offsets unavailable during rebalancing

Can anyone clarify if this is a bug in Kafka or the expected behavior? Best regards, Wouter On 30 January 2018 at 21:04, Wouter Bancken wrote: > Hi, > > I'm trying to write an external tool to monitor consumer lag on Apache > Kafka. > > For this purpose, I'm using

RE: Strange Topic ...

Hello Ted, I use Kafka_2.11-1.0.0 And I log file I only have this: [2018-01-27 23:12:09,545] INFO Shutting down the log cleaner. (kafka.log.LogCleaner) [2018-01-27 23:12:09,545] INFO [kafka-log-cleaner-thread-0]: Shutting down (kafka.log.LogCleaner) [2018-01-27 23:12:09,546] INFO

Re: Strange Topic ...

This is the topic used and created by Kafka internally to store consumer offsets while use consumer programs running. Thank you, Naresh On Sun, Feb 4, 2018 at 1:38 PM Ted Yu wrote: > Which Kafka version are you using ? > Older versions of kafka (0.10 and prior) had some

Re: Strange Topic ...

Which Kafka version are you using ? Older versions of kafka (0.10 and prior) had some bugs in the log-cleaner thread that might sometimes cause it to crash. Please check the log-cleaner.log file to see if there was some clue. Cheers On Sun, Feb 4, 2018 at 11:14 AM, adrien ruffie

Strange Topic ...

Hello all, I'm a beginner in Kafka and this morning when I try some tests and when running this following cmd: ./bin kafka-topics.sh --zookeeper localhost:2181 --describe I understand my 3 created topic like "customer-topic", "streams-plaintext-input", and "streams-wordcount-output" But I