Got it. This was indeed my problem. Thanks so much! On Fri, Nov 21, 2014 at 10:39 PM, Gwen Shapira <gshap...@cloudera.com> wrote:
> I think the issue is that you are: > " running the above snippet for every broker ... I am assuming that > item.partitionsMetadata() only returns PartitionMetadata for the partitions > this broker is responsible for " > > This is inaccurate. Each broker will check ZooKeeper for PartitionMetadata > and return information about all partitions. > You only need to check one broker. You can try other brokers if the first > broker returns an error. > > Gwen > > On Fri, Nov 21, 2014 at 10:14 PM, Rajiv Kurian <ra...@signalfuse.com> > wrote: > > > I have used the kafka-topics.sh script to create a topic with a number of > > partitions. > > > > bin/kafka-topics.sh --zookeeper myZKPath-create --topic myTopic > > --partitions 1024 --replication-factor 3 > > > > I was expecting this topic to be created with 1024 total partitions > across > > 3 of my kafka brokers. Instead it seems like it ended up creating 1024 * > 3 > > = 3072 partitions. Is this a bug? > > > > I got the number of partitions running the code from the SimpleConsumer > > example at > > > > > https://cwiki.apache.org/confluence/display/KAFKA/0.8.0+SimpleConsumer+Example > > specifically the findLeader(...) function. I just count the number of > > partitionsMetadata objects received. That seems to give me the number > > 3072. The relevant snippet looks like: > > > > List<TopicMetadata> metaData = resp.topicsMetadata(); > > for (TopicMetadata item : metaData) { > > > > for (PartitionMetadata part : item.partitionsMetadata()) { > > > > numPartitions++; > > > > } > > > > } > > > > > > Given I am running the above snippet for every broker, should I not > expect > > to see the numPartitions as 1024? I am assuming that > > item.partitionsMetadata() only returns PartitionMetadata for the > partitions > > this broker is responsible for? If not then how do I exactly figure out > > what partitions this broker is the leader for? > > > > I am using the SimpleConsumer and my logic is getting completely muddled > > because of this problem. > > > > Also is there a way through the kafka console scripts to find out the > > number of partitions for a given topic? > > > > > > Thanks, > > Rajiv > > >