Got it. This was indeed my problem. Thanks so much!

On Fri, Nov 21, 2014 at 10:39 PM, Gwen Shapira <gshap...@cloudera.com>
wrote:

> I think the issue is that you are:
> " running the above snippet for every broker ... I am assuming that
> item.partitionsMetadata() only returns PartitionMetadata for the partitions
> this broker is responsible for "
>
> This is inaccurate. Each broker will check ZooKeeper for PartitionMetadata
> and return information about all partitions.
> You only need to check one broker. You can try other brokers if the first
> broker returns an error.
>
> Gwen
>
> On Fri, Nov 21, 2014 at 10:14 PM, Rajiv Kurian <ra...@signalfuse.com>
> wrote:
>
> > I have used the kafka-topics.sh script to create a topic with a number of
> > partitions.
> >
> > bin/kafka-topics.sh  --zookeeper myZKPath-create --topic myTopic
> >  --partitions 1024 --replication-factor 3
> >
> > I was expecting this topic to be created with 1024 total partitions
> across
> > 3 of my kafka brokers. Instead it seems like it ended up creating 1024 *
> 3
> > = 3072 partitions. Is this a bug?
> >
> > I got the number of partitions running the code from the SimpleConsumer
> > example at
> >
> >
> https://cwiki.apache.org/confluence/display/KAFKA/0.8.0+SimpleConsumer+Example
> > specifically the findLeader(...) function. I just count the number of
> > partitionsMetadata objects received. That seems to give me the number
> > 3072.  The relevant snippet looks like:
> >
> > List<TopicMetadata> metaData = resp.topicsMetadata();
> > for (TopicMetadata item : metaData) {
> >
> >               for (PartitionMetadata part : item.partitionsMetadata()) {
> >
> >                     numPartitions++;
> >
> >                 }
> >
> >  }
> >
> >
> > Given I am running the above snippet for every broker, should I not
> expect
> > to see the numPartitions as 1024? I am assuming that
> > item.partitionsMetadata() only returns PartitionMetadata for the
> partitions
> > this broker is responsible for? If not then how do I exactly figure out
> > what partitions this broker is the leader for?
> >
> > I am using the SimpleConsumer and my logic is getting completely muddled
> > because of this problem.
> >
> > Also is there a way through the kafka console scripts to find out the
> > number of partitions for a given topic?
> >
> >
> > Thanks,
> > Rajiv
> >
>

Reply via email to