In 0.8, producers don't use zk. When producers encounter an error while sending data, they use a special getMetadata request to refresh the kafka cluster info from a randomly selected Kafka broker, and retry sending the data.
Thanks, Neha On Mon, Nov 19, 2012 at 12:10 PM, Jason Rosenberg <j...@squareup.com> wrote: > Are you saying that in 0.8, producers don't use zkclient? Or don't need > it? How can a producer dynamically respond to a change in the kafka > cluster without zk? > > On Mon, Nov 19, 2012 at 8:07 AM, Jun Rao <jun...@gmail.com> wrote: > >> Jae, >> >> In 0.8, producers don't need ZK client anymore. Instead, it uses a new >> getMetadata api to get topic/partition/leader information from the broker. >> Consumers still need ZK client. We plan to redesign the consumer post 0.8 >> and can keep this in mind. >> >> Thanks, >> >> Jun >> >> On Sun, Nov 18, 2012 at 10:35 PM, Bae, Jae Hyeon <metac...@gmail.com> >> wrote: >> >> > I want to suggest kafka should create only one instance of ZkClient >> > globally because ZkClient is thread safe and it will make many users >> > easily customize kafka source code for Zookeeper. >> > >> > In our company's cloud environment, it is not recommended to create >> > ZkClient from zkConnect string directly because zookeeper cluster can >> > be dynamically changing. So, I have to create ZkClient using our >> > company's own platform library. Because of this requirement, I can't >> > use kafka jar file directly. I can modify and build kafka source code >> > but I have to repeat this work whenever I update kafka version, pretty >> > annoying. >> > >> > So, my suggestion is, let me pass ZkClient outs of Producer, Consumer, >> > and Broker, as the following example. >> > >> > Producer<String, String> producer = >> > >> > >> ProducerBuilder.withZkClient(zkClient).build<String,String>(producerConfig); >> > >> > ConsumerConnector connector = >> > Consumer.withZkClient(zkClient).createJavaConsumerConnector(new >> > ConsumerConfig(consumerProps)); >> > >> > KafkaServer is a little more complicated but I believe without much >> > effort we can refactor KafkaServer to be customized with ZkClient. >> > >> > I really appreciate if this suggestion is accepted and merged to 0.8. >> > If you want me to contribute with this suggestion, please let me know >> > your opinion. If you are positive with this idea, I will contribute >> > very happily. >> > >> > Thank you >> > Best, Jae >> > >>
