Jason, In 0.8, producer doesn't use zkclient at all. You just need to set broker.list. A number of things have changed In 0.8. First, number of partitions of a topic is global in a cluster and they don't really change as new brokers are added. Second, a partition is assigned to multiple brokers for replication and one of the replicas is the leader which serves writes. When a producer starts up, it first uses the getMetadata api to figure out the replica assignment for the relevant topic/partition. It then issues producer request directly the broker where the leader resides. If the leader broker goes down, the producer gets an exception and it will re-issue the getMetadata api to obtain the information about the new leader.
Thanks, Jun On Mon, Nov 19, 2012 at 1:29 PM, Jason Rosenberg <j...@squareup.com> wrote: > Well, they do use zk though, to get the initial list of kafka nodes, and > while zk is available, presumably they do use it to keep up with the > dynamically changing set of kafka brokers, no? You are just saying that if > zk goes away, 0.8 producers can keep on producing, as long as the kafka > cluster remains stable? > > Jason > > On Mon, Nov 19, 2012 at 12:20 PM, Neha Narkhede <neha.narkh...@gmail.com > >wrote: > > > In 0.8, producers don't use zk. When producers encounter an error > > while sending data, they use a special getMetadata request to refresh > > the kafka cluster info from a randomly selected Kafka broker, and > > retry sending the data. > > > > Thanks, > > Neha > > > > On Mon, Nov 19, 2012 at 12:10 PM, Jason Rosenberg <j...@squareup.com> > > wrote: > > > Are you saying that in 0.8, producers don't use zkclient? Or don't > need > > > it? How can a producer dynamically respond to a change in the kafka > > > cluster without zk? > > > > > > On Mon, Nov 19, 2012 at 8:07 AM, Jun Rao <jun...@gmail.com> wrote: > > > > > >> Jae, > > >> > > >> In 0.8, producers don't need ZK client anymore. Instead, it uses a new > > >> getMetadata api to get topic/partition/leader information from the > > broker. > > >> Consumers still need ZK client. We plan to redesign the consumer post > > 0.8 > > >> and can keep this in mind. > > >> > > >> Thanks, > > >> > > >> Jun > > >> > > >> On Sun, Nov 18, 2012 at 10:35 PM, Bae, Jae Hyeon <metac...@gmail.com> > > >> wrote: > > >> > > >> > I want to suggest kafka should create only one instance of ZkClient > > >> > globally because ZkClient is thread safe and it will make many users > > >> > easily customize kafka source code for Zookeeper. > > >> > > > >> > In our company's cloud environment, it is not recommended to create > > >> > ZkClient from zkConnect string directly because zookeeper cluster > can > > >> > be dynamically changing. So, I have to create ZkClient using our > > >> > company's own platform library. Because of this requirement, I can't > > >> > use kafka jar file directly. I can modify and build kafka source > code > > >> > but I have to repeat this work whenever I update kafka version, > pretty > > >> > annoying. > > >> > > > >> > So, my suggestion is, let me pass ZkClient outs of Producer, > Consumer, > > >> > and Broker, as the following example. > > >> > > > >> > Producer<String, String> producer = > > >> > > > >> > > > >> > > > ProducerBuilder.withZkClient(zkClient).build<String,String>(producerConfig); > > >> > > > >> > ConsumerConnector connector = > > >> > Consumer.withZkClient(zkClient).createJavaConsumerConnector(new > > >> > ConsumerConfig(consumerProps)); > > >> > > > >> > KafkaServer is a little more complicated but I believe without much > > >> > effort we can refactor KafkaServer to be customized with ZkClient. > > >> > > > >> > I really appreciate if this suggestion is accepted and merged to > 0.8. > > >> > If you want me to contribute with this suggestion, please let me > know > > >> > your opinion. If you are positive with this idea, I will contribute > > >> > very happily. > > >> > > > >> > Thank you > > >> > Best, Jae > > >> > > > >> > > >
