In 0.8, both the broker and the consumer still need zkclient. So, a zk cluster is still needed.
Thanks, Jun On Tue, Nov 20, 2012 at 8:04 AM, Jason Rosenberg <j...@squareup.com> wrote: > Agreed, I'm not sure I understand the move away from zk. Is it still > required for consumers, and for the brokers themselves? If so, we still > need to deploy a zk cluster anyway. > > Will kafka now support coordinating 1000's of producer clients? > > Jason > > On Tue, Nov 20, 2012 at 7:54 AM, David Arthur <mum...@gmail.com> wrote: > > > > > On Nov 20, 2012, at 12:23 AM, Jun Rao wrote: > > > > > Jason, > > > > > > In 0.8, producer doesn't use zkclient at all. You just need to set > > > broker.list. > > > > This seems like a regression in functionality. For me, one of the > benefits > > of Kafka is only needing to know about ZooKeeper > > > > > A number of things have changed In 0.8. First, number of > > > partitions of a topic is global in a cluster and they don't really > change > > > as new brokers are added. Second, a partition is assigned to multiple > > > brokers for replication and one of the replicas is the leader which > > serves > > > writes. When a producer starts up, it first uses the getMetadata api to > > > figure out the replica assignment for the relevant topic/partition. It > > then > > > issues producer request directly the broker where the leader resides. > If > > > the leader broker goes down, the producer gets an exception and it will > > > re-issue the getMetadata api to obtain the information about the new > > leader. > > > > > > Thanks, > > > > > > Jun > > > > > > On Mon, Nov 19, 2012 at 1:29 PM, Jason Rosenberg <j...@squareup.com> > > wrote: > > > > > >> Well, they do use zk though, to get the initial list of kafka nodes, > and > > >> while zk is available, presumably they do use it to keep up with the > > >> dynamically changing set of kafka brokers, no? You are just saying > > that if > > >> zk goes away, 0.8 producers can keep on producing, as long as the > kafka > > >> cluster remains stable? > > >> > > >> Jason > > >> > > >> On Mon, Nov 19, 2012 at 12:20 PM, Neha Narkhede < > > neha.narkh...@gmail.com > > >>> wrote: > > >> > > >>> In 0.8, producers don't use zk. When producers encounter an error > > >>> while sending data, they use a special getMetadata request to refresh > > >>> the kafka cluster info from a randomly selected Kafka broker, and > > >>> retry sending the data. > > >>> > > >>> Thanks, > > >>> Neha > > >>> > > >>> On Mon, Nov 19, 2012 at 12:10 PM, Jason Rosenberg <j...@squareup.com> > > >>> wrote: > > >>>> Are you saying that in 0.8, producers don't use zkclient? Or don't > > >> need > > >>>> it? How can a producer dynamically respond to a change in the kafka > > >>>> cluster without zk? > > >>>> > > >>>> On Mon, Nov 19, 2012 at 8:07 AM, Jun Rao <jun...@gmail.com> wrote: > > >>>> > > >>>>> Jae, > > >>>>> > > >>>>> In 0.8, producers don't need ZK client anymore. Instead, it uses a > > new > > >>>>> getMetadata api to get topic/partition/leader information from the > > >>> broker. > > >>>>> Consumers still need ZK client. We plan to redesign the consumer > post > > >>> 0.8 > > >>>>> and can keep this in mind. > > >>>>> > > >>>>> Thanks, > > >>>>> > > >>>>> Jun > > >>>>> > > >>>>> On Sun, Nov 18, 2012 at 10:35 PM, Bae, Jae Hyeon < > metac...@gmail.com > > > > > >>>>> wrote: > > >>>>> > > >>>>>> I want to suggest kafka should create only one instance of > ZkClient > > >>>>>> globally because ZkClient is thread safe and it will make many > users > > >>>>>> easily customize kafka source code for Zookeeper. > > >>>>>> > > >>>>>> In our company's cloud environment, it is not recommended to > create > > >>>>>> ZkClient from zkConnect string directly because zookeeper cluster > > >> can > > >>>>>> be dynamically changing. So, I have to create ZkClient using our > > >>>>>> company's own platform library. Because of this requirement, I > can't > > >>>>>> use kafka jar file directly. I can modify and build kafka source > > >> code > > >>>>>> but I have to repeat this work whenever I update kafka version, > > >> pretty > > >>>>>> annoying. > > >>>>>> > > >>>>>> So, my suggestion is, let me pass ZkClient outs of Producer, > > >> Consumer, > > >>>>>> and Broker, as the following example. > > >>>>>> > > >>>>>> Producer<String, String> producer = > > >>>>>> > > >>>>>> > > >>>>> > > >>> > > >> > > > ProducerBuilder.withZkClient(zkClient).build<String,String>(producerConfig); > > >>>>>> > > >>>>>> ConsumerConnector connector = > > >>>>>> Consumer.withZkClient(zkClient).createJavaConsumerConnector(new > > >>>>>> ConsumerConfig(consumerProps)); > > >>>>>> > > >>>>>> KafkaServer is a little more complicated but I believe without > much > > >>>>>> effort we can refactor KafkaServer to be customized with ZkClient. > > >>>>>> > > >>>>>> I really appreciate if this suggestion is accepted and merged to > > >> 0.8. > > >>>>>> If you want me to contribute with this suggestion, please let me > > >> know > > >>>>>> your opinion. If you are positive with this idea, I will > contribute > > >>>>>> very happily. > > >>>>>> > > >>>>>> Thank you > > >>>>>> Best, Jae > > >>>>>> > > >>>>> > > >>> > > >> > > > > >
