If I understand correctly (and I'll be happy if someone who knows more will jump in and correct me):
The Sync/Async part is not between the producer and the broker. Its between you and the producer. The Sync producer takes your message and immediately contacts the broker, sends the message, either wait for acks or not and returns. The Async producer takes your message and immediately returns. The producer will send the message to the broker at some time later, batching multiple requests for efficiency and throughput. So yeah, I think you got it mostly right. Just note that the producer doesn't wait on .send, the producer executes the send - either returning immediately (if async) or when it managed to contact the broker (if sync). Gwen On Fri, Oct 17, 2014 at 4:38 PM, Mohit Anchlia <mohitanch...@gmail.com> wrote: > My understanding of sync is that producer waits on .send until Kafka > receives the message. And async means it just dispatches the message > without any gurantees that message is delivered. Did I get that part right? > On Fri, Oct 17, 2014 at 1:28 PM, Gwen Shapira <gshap...@cloudera.com> wrote: > >> Sorry if I'm confusing you :) >> >> Kafka 0.8.1.1 has two producers sync and async. You are using the sync >> producer without waiting for acks. I hope this helps? >> >> Regardless, did you check if the partition got created? are you able >> to produce messages? are you able to consume them? >> >> Gwen >> >> On Fri, Oct 17, 2014 at 4:13 PM, Mohit Anchlia <mohitanch...@gmail.com> >> wrote: >> > Still don't understand the difference. If it's not waiting for the ack >> then >> > doesn't it make async? >> > On Fri, Oct 17, 2014 at 12:55 PM, <gshap...@cloudera.com> wrote: >> > >> >> Its using the sync producer without waiting for any broker to >> acknowledge >> >> the write. This explains the lack of errors you are seeing. >> >> >> >> — >> >> Sent from Mailbox >> >> >> >> On Fri, Oct 17, 2014 at 3:15 PM, Mohit Anchlia <mohitanch...@gmail.com> >> >> wrote: >> >> >> >> > Little confused :) From one of the examples I am using property >> >> > request.required.acks=0, >> >> > I thought this sets the producer to be async? >> >> > On Fri, Oct 17, 2014 at 11:59 AM, Gwen Shapira <gshap...@cloudera.com >> > >> >> > wrote: >> >> >> 0.8.1.1 producer is Sync by default, and you can set producer.type to >> >> >> async if needed. >> >> >> >> >> >> On Fri, Oct 17, 2014 at 2:57 PM, Mohit Anchlia < >> mohitanch...@gmail.com> >> >> >> wrote: >> >> >> > Thanks! How can I tell if I am using async producer? I thought all >> the >> >> >> > sends are async in nature >> >> >> > On Fri, Oct 17, 2014 at 11:44 AM, Gwen Shapira < >> gshap...@cloudera.com >> >> > >> >> >> > wrote: >> >> >> > >> >> >> >> If you have "auto.create.topics.enable" set to "true" (default), >> >> >> >> producing to a topic creates it. >> >> >> >> >> >> >> >> Its a bit tricky because the "send" that creates the topic can >> fail >> >> >> >> with "leader not found" or similar issue. retrying few times will >> >> >> >> eventually succeed as the topic gets created and the leader gets >> >> >> >> elected. >> >> >> >> >> >> >> >> Is it possible that you are not getting errors because you are >> using >> >> >> >> async producer? >> >> >> >> >> >> >> >> Also "no messages are delivered" can have many causes. Check if >> the >> >> >> >> topic exists using: >> >> >> >> bin/kafka-topics.sh --list --zookeeper localhost:2181 >> >> >> >> >> >> >> >> Perhaps the topic was created and the issue is elsewhere (the >> >> consumer >> >> >> >> is a usual suspect! perhaps look in the FAQ for tips with that >> issue) >> >> >> >> >> >> >> >> Gwen >> >> >> >> >> >> >> >> On Fri, Oct 17, 2014 at 12:56 PM, Mohit Anchlia < >> >> mohitanch...@gmail.com >> >> >> > >> >> >> >> wrote: >> >> >> >> > Is Kafka supposed to throw exception if topic doesn't exist? It >> >> >> appears >> >> >> >> > that there is no exception thrown even though no messages are >> >> >> delivered >> >> >> >> and >> >> >> >> > there are errors logged in Kafka logs. >> >> >> >> >> >> >> >> >> >>
