Agreed, I'm not sure I understand the move away from zk. Is it still required for consumers, and for the brokers themselves? If so, we still need to deploy a zk cluster anyway.
Will kafka now support coordinating 1000's of producer clients? Jason On Tue, Nov 20, 2012 at 7:54 AM, David Arthur <mum...@gmail.com> wrote: > > On Nov 20, 2012, at 12:23 AM, Jun Rao wrote: > > > Jason, > > > > In 0.8, producer doesn't use zkclient at all. You just need to set > > broker.list. > > This seems like a regression in functionality. For me, one of the benefits > of Kafka is only needing to know about ZooKeeper > > > A number of things have changed In 0.8. First, number of > > partitions of a topic is global in a cluster and they don't really change > > as new brokers are added. Second, a partition is assigned to multiple > > brokers for replication and one of the replicas is the leader which > serves > > writes. When a producer starts up, it first uses the getMetadata api to > > figure out the replica assignment for the relevant topic/partition. It > then > > issues producer request directly the broker where the leader resides. If > > the leader broker goes down, the producer gets an exception and it will > > re-issue the getMetadata api to obtain the information about the new > leader. > > > > Thanks, > > > > Jun > > > > On Mon, Nov 19, 2012 at 1:29 PM, Jason Rosenberg <j...@squareup.com> > wrote: > > > >> Well, they do use zk though, to get the initial list of kafka nodes, and > >> while zk is available, presumably they do use it to keep up with the > >> dynamically changing set of kafka brokers, no? You are just saying > that if > >> zk goes away, 0.8 producers can keep on producing, as long as the kafka > >> cluster remains stable? > >> > >> Jason > >> > >> On Mon, Nov 19, 2012 at 12:20 PM, Neha Narkhede < > neha.narkh...@gmail.com > >>> wrote: > >> > >>> In 0.8, producers don't use zk. When producers encounter an error > >>> while sending data, they use a special getMetadata request to refresh > >>> the kafka cluster info from a randomly selected Kafka broker, and > >>> retry sending the data. > >>> > >>> Thanks, > >>> Neha > >>> > >>> On Mon, Nov 19, 2012 at 12:10 PM, Jason Rosenberg <j...@squareup.com> > >>> wrote: > >>>> Are you saying that in 0.8, producers don't use zkclient? Or don't > >> need > >>>> it? How can a producer dynamically respond to a change in the kafka > >>>> cluster without zk? > >>>> > >>>> On Mon, Nov 19, 2012 at 8:07 AM, Jun Rao <jun...@gmail.com> wrote: > >>>> > >>>>> Jae, > >>>>> > >>>>> In 0.8, producers don't need ZK client anymore. Instead, it uses a > new > >>>>> getMetadata api to get topic/partition/leader information from the > >>> broker. > >>>>> Consumers still need ZK client. We plan to redesign the consumer post > >>> 0.8 > >>>>> and can keep this in mind. > >>>>> > >>>>> Thanks, > >>>>> > >>>>> Jun > >>>>> > >>>>> On Sun, Nov 18, 2012 at 10:35 PM, Bae, Jae Hyeon <metac...@gmail.com > > > >>>>> wrote: > >>>>> > >>>>>> I want to suggest kafka should create only one instance of ZkClient > >>>>>> globally because ZkClient is thread safe and it will make many users > >>>>>> easily customize kafka source code for Zookeeper. > >>>>>> > >>>>>> In our company's cloud environment, it is not recommended to create > >>>>>> ZkClient from zkConnect string directly because zookeeper cluster > >> can > >>>>>> be dynamically changing. So, I have to create ZkClient using our > >>>>>> company's own platform library. Because of this requirement, I can't > >>>>>> use kafka jar file directly. I can modify and build kafka source > >> code > >>>>>> but I have to repeat this work whenever I update kafka version, > >> pretty > >>>>>> annoying. > >>>>>> > >>>>>> So, my suggestion is, let me pass ZkClient outs of Producer, > >> Consumer, > >>>>>> and Broker, as the following example. > >>>>>> > >>>>>> Producer<String, String> producer = > >>>>>> > >>>>>> > >>>>> > >>> > >> > ProducerBuilder.withZkClient(zkClient).build<String,String>(producerConfig); > >>>>>> > >>>>>> ConsumerConnector connector = > >>>>>> Consumer.withZkClient(zkClient).createJavaConsumerConnector(new > >>>>>> ConsumerConfig(consumerProps)); > >>>>>> > >>>>>> KafkaServer is a little more complicated but I believe without much > >>>>>> effort we can refactor KafkaServer to be customized with ZkClient. > >>>>>> > >>>>>> I really appreciate if this suggestion is accepted and merged to > >> 0.8. > >>>>>> If you want me to contribute with this suggestion, please let me > >> know > >>>>>> your opinion. If you are positive with this idea, I will contribute > >>>>>> very happily. > >>>>>> > >>>>>> Thank you > >>>>>> Best, Jae > >>>>>> > >>>>> > >>> > >> > >
