For reference, and to close this out: It turns out there is another way for this to happen. Given a stable Kafka cluster, if the producer that first sends messages for a new topic does not send "enough" messages, or it's session does not last "long enough", the bootstrapping process will not distribute the topic and partitions to all brokers.
I can't nail down the definition of "enough messages" or a session that is "long enough", but the point is that this can happen even with a stable set of brokers. I've updated the JIRA issue. Thanks, Matt -----Original Message----- From: Jun Rao [mailto:jun...@gmail.com] Sent: Friday, February 17, 2012 3:03 PM To: kafka-users@incubator.apache.org Subject: Re: Only one broker has partitions Thanks, Bateman. Jun On Fri, Feb 17, 2012 at 2:58 PM, Bateman, Matt <mabate...@ebay.com> wrote: > Hi Jun, > > https://issues.apache.org/jira/browse/KAFKA-278 > > Thanks, > > Matt > > -----Original Message----- > From: Jun Rao [mailto:jun...@gmail.com] > Sent: Friday, February 17, 2012 2:23 PM > To: kafka-users@incubator.apache.org > Subject: Re: Only one broker has partitions > > I see the problem. This is a bug in Kafka. Basically, for a new topic, > we bootstrap using all existing brokers. However, if a topic already > exists on some brokers, we never bootstrap again, which means new > brokers will be ignored. Could you file a jira for that? For now, you > have to manually create the topic on the new brokers (e.g., by sending > some data to the new broker directly). > > Thanks, > > Jun > > > On Fri, Feb 17, 2012 at 1:40 PM, Bateman, Matt <mabate...@ebay.com> wrote: > > > Hi Jun, > > > > Thanks for the details. What sort of time frame is eventually? > > > > In the case of a topic that exists before nodes are added, will the > > other nodes ever get that topic? We've been running all three for > > about a week now it's still only on our 0 broker. > > > > Thanks, > > > > Matt > > > > -----Original Message----- > > From: Jun Rao [mailto:jun...@gmail.com] > > Sent: Friday, February 17, 2012 11:19 AM > > To: kafka-users@incubator.apache.org > > Subject: Re: Only one broker has partitions > > > > Matt, > > > > The behavior is a bit weird for a new topic. Initially, non of the > > brokers has the topic. So, the producer simply assumes there is 1 > > partition per broker. Once a topic is created in a broker, the > > broker registers #partitions for that topic in ZK. Then the producer > > will pick up the new #partitions for that broker and refresh its > > list of > available partitions. > > Eventually, the topic will be created on every broker and the > > producer should see a total of 60 partitions. > > > > Thanks, > > > > Jun > > > > > > On Fri, Feb 17, 2012 at 11:05 AM, Bateman, Matt <mabate...@ebay.com> > > wrote: > > > > > We're using a single topic ("test-topic") in our QA setup. > > > Originally we were using only one broker (id 0) in or testing > environment. > > > Recently we added two more (id 2 and 3) for a total of 3 in our > > > cluster. ZooKeeper shows that the topic and partitions are > > > registered by > > broker 0 as expected. > > > They do not show for brokers 2 and 3. I'm guessing that if a > > > broker is not registered with ZooKeeper when a producer sends the > > > first message for a topic, it will never process messages for that topic? > > > > > > As a test, I sent messages into the cluster for a new topic. The > > > producer logs show > > > > > > [2012-02-17 10:31:12,698] DEBUG Getting the number of broker > > > partitions registered for topic: cluster-test > > > (kafka.producer.Producer) > > > [2012-02-17 10:31:12,698] DEBUG Currently, no brokers are > > > registered under > > > topic: cluster-test (kafka.producer.ZKBrokerPartitionInfo) > > > [2012-02-17 10:31:12,698] DEBUG Bootstrapping topic: cluster-test > > > with available brokers in the cluster with default number of > > > partitions = 1 > > > (kafka.producer.ZKBrokerPartitionInfo) > > > [2012-02-17 10:31:12,705] DEBUG Adding following broker id, > > > partition id for NEW topic: cluster-test=TreeSet(0-0, 2-0, 3-0) > > > (kafka.producer.ZKBrokerPartitionInfo) > > > [2012-02-17 10:31:12,705] DEBUG Broker partitions registered for topic: > > > cluster-test = List(0-0, 2-0, 3-0) (kafka.producer.Producer) > > > > > > This seems logical, all brokers are mentioned. Then I see the > > > following (each broker is configured to use 20 partitions): > > > > > > [2012-02-17 10:31:12,840] DEBUG Sending message to broker 2 > > > (kafka.producer.ProducerPool) > > > [2012-02-17 10:31:12,953] DEBUG [BrokerTopicsListener] List of > > > topics changed at /brokers/topics Updated topics -> [topic1, > > > topic2, cluster-test, topic3, test-topic, topic4] > > > (kafka.producer.ZKBrokerPartitionInfo$BrokerTopicsListener) > > > [2012-02-17 10:31:12,953] DEBUG [BrokerTopicsListener] Old list of > > topics: > > > Set(topic1, topic3, topic2, test-topic, topic4) > > > (kafka.producer.ZKBrokerPartitionInfo$BrokerTopicsListener) > > > [2012-02-17 10:31:12,963] DEBUG [BrokerTopicsListener] Updated > > > list of > > > topics: Set(topic4, test-topic, topic2, topic3, cluster-test, > > > topic1) > > > (kafka.producer.ZKBrokerPartitionInfo$BrokerTopicsListener) > > > [2012-02-17 10:31:12,963] DEBUG [BrokerTopicsListener] List of > > > newly registered topics: Set(cluster-test) > > > (kafka.producer.ZKBrokerPartitionInfo$BrokerTopicsListener) > > > [2012-02-17 10:31:12,978] DEBUG [BrokerTopicsListener] Currently > > > registered list of brokers for topic: cluster-test are Buffer(2) > > > (kafka.producer.ZKBrokerPartitionInfo$BrokerTopicsListener) > > > [2012-02-17 10:31:12,978] DEBUG [BrokerTopicsListener] > > > Unregistered list of brokers for topic: cluster-test are > > > TreeSet(0-0, 2-0, 3-0) > > > (kafka.producer.ZKBrokerPartitionInfo$BrokerTopicsListener) > > > [2012-02-17 10:31:12,980] DEBUG [BrokerTopicsListener] List of > > > broker partitions for topic: cluster-test are TreeSet(0-0, 2-0, > > > 2-1, 2-2, 2-3, 2-4, 2-5, 2-6, 2-7, 2-8, 2-9, 2-10, 2-11, 2-12, > > > 2-13, 2-14, 2-15, 2-16, 2-17, 2-18, 2-19, 3-0) > > > (kafka.producer.ZKBrokerPartitionInfo$BrokerTopicsListener) > > > [2012-02-17 10:31:13,841] DEBUG Getting the number of broker > > > partitions registered for topic: cluster-test > > > (kafka.producer.Producer) > > > [2012-02-17 10:31:13,841] DEBUG Broker partitions registered for topic: > > > cluster-test = List(0-0, 2-0, 2-1, 2-2, 2-3, 2-4, 2-5, 2-6, 2-7, > > > 2-8, 2-9, 2-10, 2-11, 2-12, 2-13, 2-14, 2-15, 2-16, 2-17, 2-18, > > > 2-19, 3-0) > > > (kafka.producer.Producer) > > > > > > This seems strange, shouldn't each broker host each partition for > > > the new "cluster-topic"? > > > > > > I then see the producer debug messages showing messages sent to > > > broker > 2. > > > > > > What am I missing here? > > > > > > Thanks, > > > > > > Matt > > > > > > -----Original Message----- > > > From: Jun Rao [mailto:jun...@gmail.com] > > > Sent: Thursday, February 16, 2012 6:08 PM > > > To: kafka-users@incubator.apache.org > > > Subject: Re: Only one broker has partitions > > > > > > Try enabling debug level logging in Producer class. It will show > > > which broker the partitioner selects. > > > > > > Thanks, > > > > > > Jun > > > > > > On Thu, Feb 16, 2012 at 2:48 PM, Bateman, Matt > > > <mabate...@ebay.com> > > wrote: > > > > > > > Hi All, > > > > > > > > I feel like this was discussed at some point but couldn't find > > > > it in the docs or the mailing list archives, hopefully this > > > > isn't > redundant. > > > > > > > > We are running 3 version 0.7 brokers using an ensemble of 3 > > > > ZooKeeper nodes. We use a ZooKeeper connect string in the > > > > producers > > and consumers. > > > > ZooKeeper is enabled on each broker. We use the default > > > > partitioner in the producer supplying it an integer. Each broker > > > > has > a unique Id. > > > > > > > > However, only one broker has any log files. The log from that > > > > machine shows it registering the topics in ZooKeeper. The other > > > > brokers don't have any log files. > > > > > > > > Have a I missed a basic configuration setting or misunderstood > > > > something operational? The only slight mismatch I see is we run > > > > ZooKeeper 3.3.4 but it seems like 3.3.3 is being used by 0.7. > > > > > > > > Thanks, > > > > > > > > Matt > > > > > > > > > >
