Re: How to verify offsets topic exists?

2015-10-09 Thread Stevo Slavić
If I'm not mistaken, replication factor of a topic does not get stored in ZK - on creation replication factor gets translated to topic replica assignment which gets stored in ZK. Again, please correct me if wrong - it seems that only during topic creation it is verified that all replicas in

Re: How to verify offsets topic exists?

2015-10-08 Thread Stevo Slavić
There's another related bug - triggering offsets topic creation through requesting metadata about that topic does not work in case of single broker clean (no topics created yet) Kafka cluster running. In that case sequence returned by KafkaApis.getAliveBrokers is empty, and

Re: How to verify offsets topic exists?

2015-10-06 Thread Stevo Slavić
Thanks Grant for quick reply! I've used AdminUtils.topicExists("__consumer_offsets") check and even 10sec after Kafka broker startup, the check fails. When, on which event, does this internal topic get created? Is there some broker config property preventing it from being created? Does one have

Re: How to verify offsets topic exists?

2015-10-06 Thread Stevo Slavić
Debugged, and found in KafkaApis.handleConsumerMetadataRequest that consumer offsets topic gets created on first lookup of offsets topic metadata, even when auto topic creation is disabled. In that method there is following call: // get metadata (and create the topic if necessary) val

How to verify offsets topic exists?

2015-10-05 Thread Stevo Slavić
Hello Apache Kafka community, In my integration tests, with single 0.8.2.2 broker, for newly created topic with single partition, after determining through topic metadata request that partition has lead broker assigned, when I try to reset offset for given consumer group, I first try to discover

Re: How to verify offsets topic exists?

2015-10-05 Thread Grant Henke
Hi Stevo, There are a couple of options to verify the topic exists: 1. Consume from a topic with "offsets.storage=kafka". If its not created already, this should create it. 2. List and describe the topic using the Kafka topics script. Ex: bin/kafka-topics.sh --zookeeper localhost:2181