I meant the producer. ( i.e. application using the producer api to push messages into kafka ) .
On Wed, May 13, 2015 at 10:20 PM, Mayuresh Gharat < gharatmayures...@gmail.com> wrote: > By application rebooting, do you mean you bounce the brokers? > > Thanks, > > Mayuresh > > On Wed, May 13, 2015 at 4:06 AM, Mohit Gupta < > success.mohit.gu...@gmail.com> > wrote: > > > Thanks Jiangjie. This is helpful. > > > > Adding to what you have mentioned, I can think of one more scenario which > > may not be very rare. > > Say, the application is rebooted and the Kafka brokers registered in the > > producer are not reachable ( could be due to network issues or those > > brokers are actually down ). Since, no metadata is available the send > will > > block. Right? > > > > On Wed, May 13, 2015 at 10:51 AM, Jiangjie Qin <j...@linkedin.com.invalid > > > > wrote: > > > > > > > > Application will not block on each metadata refresh or metadata is > > > expired. > > > Application will only be blocked when > > > 1. It sends the first message to a topic (only for that single > message), > > or > > > 2. The topic has been deleted from broker thus refreshed metadata loses > > > the topic info (which is pretty rare). > > > > > > So I think the async here might mean a little bit different. It means > > when > > > you send first message to a topic, you wait till you know the topic > > exist, > > > after that point it is async. > > > It is very low chance that your application will block on send. If it > is > > > then something probably really went wrong and needs immediate > attention. > > > > > > Thanks. > > > > > > Jiangjie (Becket) Qin > > > > > > On 5/12/15, 5:08 PM, "Rendy Bambang Junior" <rendy.b.jun...@gmail.com> > > > wrote: > > > > > > >Thank you for the clarification. > > > > > > > >I think I agree with Mohit. Sometime blocking on logging is not > > acceptable > > > >by nature of application who uses kafka. > > > > > > > >Yes it is not blocking when metadata is still available. But > application > > > >will be blocked once metada is expired. > > > > > > > >It might be handled by application, by implementing async call when do > > > >send() and manage buffer and async timeout internally, but it makes > > async > > > >feature in kafka producer has less meaning. > > > > > > > >Sorry if my understanding is incorrect. > > > > > > > >Rendy > > > >On May 13, 2015 6:59 AM, "Jiangjie Qin" <j...@linkedin.com.invalid> > > > wrote: > > > > > > > >> Send() will only block if the metadata is *not available* for the > > topic. > > > >> It won't block if metadata there is stale. The metadata refresh is > > async > > > >> to send(). However, if you send the message to a topic for the first > > > >>time, > > > >> send() will trigger a metadata refresh and block until it has > metadata > > > >>for > > > >> that topic. > > > >> > > > >> Jiangjie (Becket) Qin > > > >> > > > >> On 5/12/15, 12:58 PM, "Magnus Edenhill" <mag...@edenhill.se> wrote: > > > >> > > > >> >I completely agree with Mohit, an application should not have to > know > > > >>or > > > >> >care about > > > >> >producer implementation internals. > > > >> >Given a message and its delivery constraints (produce retry count > and > > > >> >timeout) the producer > > > >> >should hide any temporal failures until the message is succesfully > > > >> >delivered, a permanent > > > >> >error is encountered or the constraints are hit. > > > >> >This should also include internal start up sequencing, such as > > metadata > > > >> >retrieval. > > > >> > > > > >> > > > > >> > > > > >> >2015-05-12 21:22 GMT+02:00 Mohit Gupta < > > success.mohit.gu...@gmail.com > > > >: > > > >> > > > > >> >> I could not follow the reasoning behind blocking the send method > if > > > >>the > > > >> >> metadata is not up-to-date. Though, I see that it as per design, > it > > > >> >> requires the metadata to batch the message into appropriate > > > >> >>topicPartition > > > >> >> queue. Also, if the metadata could not be updated in the > specified > > > >> >> interval, it throws an exception and the message is not queued to > > be > > > >> >> retried once the brokers are up. > > > >> >> > > > >> >> Should it not be that messages are buffered in another queue ( > > up-to > > > >>a > > > >> >> limit ) if the brokers are down and retried later? > > > >> >> Is it not a general use case to require producer to be > asynchronous > > > >>in > > > >> >>all > > > >> >> the scenarios? > > > >> >> > > > >> >> > > > >> >> On Tue, May 12, 2015 at 10:54 PM, Mayuresh Gharat < > > > >> >> gharatmayures...@gmail.com> wrote: > > > >> >> > > > >> >> > The way it works I suppose is that, the producer will do > > > >> >>fetchMetadata, > > > >> >> if > > > >> >> > the last fetched metadata is stale (the refresh interval has > > > >>expired) > > > >> >>or > > > >> >> if > > > >> >> > it is not able to send data to a particular broker in its > current > > > >> >> metadata > > > >> >> > (This might happen in some cases like if the leader moves). > > > >> >> > > > > >> >> > It cannot produce without having the right metadata. > > > >> >> > > > > >> >> > Thanks, > > > >> >> > > > > >> >> > Mayuresh > > > >> >> > > > > >> >> > On Tue, May 12, 2015 at 10:09 AM, Jiangjie Qin > > > >> >><j...@linkedin.com.invalid > > > >> >> > > > > >> >> > wrote: > > > >> >> > > > > >> >> > > That¹s right. Send() will first try to get metadata of a > topic, > > > >>that > > > >> >> is a > > > >> >> > > blocking operation. > > > >> >> > > > > > >> >> > > On 5/12/15, 2:48 AM, "Rendy Bambang Junior" > > > >> >><rendy.b.jun...@gmail.com> > > > >> >> > > wrote: > > > >> >> > > > > > >> >> > > >Hi, sorry if my understanding is incorrect. > > > >> >> > > > > > > >> >> > > >I am integrating kafka producer with application, when i try > > to > > > >> >> shutdown > > > >> >> > > >all kafka broker (preparing for prod env) I notice that > 'send' > > > >> >>method > > > >> >> is > > > >> >> > > >blocking. > > > >> >> > > > > > > >> >> > > >Is new producer fetch metadata not async? > > > >> >> > > > > > > >> >> > > >Rendy > > > >> >> > > > > > >> >> > > > > > >> >> > > > > >> >> > > > > >> >> > -- > > > >> >> > -Regards, > > > >> >> > Mayuresh R. Gharat > > > >> >> > (862) 250-7125 > > > >> >> > > > > >> >> > > > >> >> > > > >> >> > > > >> >> -- > > > >> >> Best Regards, > > > >> >> > > > >> >> Mohit Gupta > > > >> >> > > > >> > > > >> > > > > > > > > > > > > -- > > Best Regards, > > > > Mohit Gupta > > > > > > -- > -Regards, > Mayuresh R. Gharat > (862) 250-7125 > -- Best Regards, Mohit Gupta
