There's some good advice here: https://kafka.apache.org/081/ops.html
And you may enjoy this post too: http://blog.liveramp.com/2013/04/08/kafka-0-8-producer-performance-2/ On Fri, Oct 17, 2014 at 5:52 PM, Mohit Anchlia <mohitanch...@gmail.com> wrote: > Thanks for the info. I see there are tons of parameters but is there a > place that lists some important performance specific parameters? > On Fri, Oct 17, 2014 at 2:43 PM, Gwen Shapira <gshap...@cloudera.com> wrote: > >> If I understand correctly (and I'll be happy if someone who knows more >> will jump in and correct me): >> >> The Sync/Async part is not between the producer and the broker. Its >> between you and the producer. The Sync producer takes your message and >> immediately contacts the broker, sends the message, either wait for >> acks or not and returns. The Async producer takes your message and >> immediately returns. The producer will send the message to the broker >> at some time later, batching multiple requests for efficiency and >> throughput. >> >> So yeah, I think you got it mostly right. Just note that the producer >> doesn't wait on .send, the producer executes the send - either >> returning immediately (if async) or when it managed to contact the >> broker (if sync). >> >> Gwen >> >> On Fri, Oct 17, 2014 at 4:38 PM, Mohit Anchlia <mohitanch...@gmail.com> >> wrote: >> > My understanding of sync is that producer waits on .send until Kafka >> > receives the message. And async means it just dispatches the message >> > without any gurantees that message is delivered. Did I get that part >> right? >> > On Fri, Oct 17, 2014 at 1:28 PM, Gwen Shapira <gshap...@cloudera.com> >> wrote: >> > >> >> Sorry if I'm confusing you :) >> >> >> >> Kafka 0.8.1.1 has two producers sync and async. You are using the sync >> >> producer without waiting for acks. I hope this helps? >> >> >> >> Regardless, did you check if the partition got created? are you able >> >> to produce messages? are you able to consume them? >> >> >> >> Gwen >> >> >> >> On Fri, Oct 17, 2014 at 4:13 PM, Mohit Anchlia <mohitanch...@gmail.com> >> >> wrote: >> >> > Still don't understand the difference. If it's not waiting for the ack >> >> then >> >> > doesn't it make async? >> >> > On Fri, Oct 17, 2014 at 12:55 PM, <gshap...@cloudera.com> wrote: >> >> > >> >> >> Its using the sync producer without waiting for any broker to >> >> acknowledge >> >> >> the write. This explains the lack of errors you are seeing. >> >> >> >> >> >> — >> >> >> Sent from Mailbox >> >> >> >> >> >> On Fri, Oct 17, 2014 at 3:15 PM, Mohit Anchlia < >> mohitanch...@gmail.com> >> >> >> wrote: >> >> >> >> >> >> > Little confused :) From one of the examples I am using property >> >> >> > request.required.acks=0, >> >> >> > I thought this sets the producer to be async? >> >> >> > On Fri, Oct 17, 2014 at 11:59 AM, Gwen Shapira < >> gshap...@cloudera.com >> >> > >> >> >> > wrote: >> >> >> >> 0.8.1.1 producer is Sync by default, and you can set >> producer.type to >> >> >> >> async if needed. >> >> >> >> >> >> >> >> On Fri, Oct 17, 2014 at 2:57 PM, Mohit Anchlia < >> >> mohitanch...@gmail.com> >> >> >> >> wrote: >> >> >> >> > Thanks! How can I tell if I am using async producer? I thought >> all >> >> the >> >> >> >> > sends are async in nature >> >> >> >> > On Fri, Oct 17, 2014 at 11:44 AM, Gwen Shapira < >> >> gshap...@cloudera.com >> >> >> > >> >> >> >> > wrote: >> >> >> >> > >> >> >> >> >> If you have "auto.create.topics.enable" set to "true" >> (default), >> >> >> >> >> producing to a topic creates it. >> >> >> >> >> >> >> >> >> >> Its a bit tricky because the "send" that creates the topic can >> >> fail >> >> >> >> >> with "leader not found" or similar issue. retrying few times >> will >> >> >> >> >> eventually succeed as the topic gets created and the leader >> gets >> >> >> >> >> elected. >> >> >> >> >> >> >> >> >> >> Is it possible that you are not getting errors because you are >> >> using >> >> >> >> >> async producer? >> >> >> >> >> >> >> >> >> >> Also "no messages are delivered" can have many causes. Check if >> >> the >> >> >> >> >> topic exists using: >> >> >> >> >> bin/kafka-topics.sh --list --zookeeper localhost:2181 >> >> >> >> >> >> >> >> >> >> Perhaps the topic was created and the issue is elsewhere (the >> >> >> consumer >> >> >> >> >> is a usual suspect! perhaps look in the FAQ for tips with that >> >> issue) >> >> >> >> >> >> >> >> >> >> Gwen >> >> >> >> >> >> >> >> >> >> On Fri, Oct 17, 2014 at 12:56 PM, Mohit Anchlia < >> >> >> mohitanch...@gmail.com >> >> >> >> > >> >> >> >> >> wrote: >> >> >> >> >> > Is Kafka supposed to throw exception if topic doesn't exist? >> It >> >> >> >> appears >> >> >> >> >> > that there is no exception thrown even though no messages are >> >> >> >> delivered >> >> >> >> >> and >> >> >> >> >> > there are errors logged in Kafka logs. >> >> >> >> >> >> >> >> >> >> >> >> >> >> >>
