At LinkedIn we've run into some of these problems. We're slowly making our way towards a solution. It's a little odd in the sense that while it's possible for some of our clients to block if the cluster is down, sometimes that is the behavior we need. For us (in production) a cluster down or unresponsive is something we can't tolerate, so our clients rarely encounter that behavior because we deal with these situations immediately.
Recently the design choices of poll(timeout) in new consumer have surfaced again. Even if blocking rarely happens, the correct behavior (imo) is for poll(timeout) to return within the timeout, even when it's waiting for a broker to come back up or for metadata to be retrieved. How this is done is still something we don't have agreement on. As soon as we have something to contribute we'll send it to the list. If others have thoughts, we would welcome those as well. Nacho On Mon, Oct 31, 2016 at 2:21 PM, Andrey Dyachkov <andrey.dyach...@gmail.com> wrote: > Hi Nacho, > > yes, exactly. > > On Mon, 31 Oct 2016 at 22:10 Nacho Solis <nso...@linkedin.com.invalid> > wrote: > > > Hi Andrew. > > > > Is this what you're saying: > > - sometimes you get stuck (like a blocking call) when you call some > > function in the kafka client > > - you can go around this case (by wrapping it) such that if the kafka > > client call gets stuck your software doesn't get stuck > > - you're wondering why the kafka client is not more resilient to the > broker > > being unavailable/down > > > > Is this correct? > > > > Nacho > > > > On Mon, Oct 31, 2016 at 1:54 PM, Andrey Dyachkov < > > andrey.dyach...@gmail.com> > > wrote: > > > > > Hi Becket, > > > > > > I asked the same question in the user mailing list and we had a small > > > discussion there. > > > > > > We have default timeout for producer and consumer it is 30 secs. > > > If Kafka is not available we will wait 30 secs on each request. > > > Decreasing timeout won't help much because we have a lot requests > > > but circuiting them will help us tremendous. We could implement such > > > logic before actually calling Kafka client but there is problem what > the > > > trigger is to open the circuit. I thought about circuiting by > unavailable > > > broker > > > but it is quite an expansive operation to figure out in the client's > code > > > on the flip side Kaka client already does all the job to make out the > > > broker. > > > > > > For us very important to stay live even if Kafka is unavailable (it > > > happens) that's why I think avoiding cascading issues will be very > > useful. > > > > > > The circuit breaker here is a thing which can stop requests to Kafka > for > > > certain period of time until a specific topic/partition/broker is not > > > available. > > > > > > On Mon, 31 Oct 2016 at 21:38 Becket Qin <becket....@gmail.com> wrote: > > > > > > Hi Andrey, > > > > > > I am not sure I understand the question "why doesn't Kafka client have > > > the solution > > > for this kind of problems?" Can you elaborate a little on what kind of > > > problems are you referring to? And what do you mean by "circuit > breaker" > > > here? > > > > > > Jiangjie (Becket) Qin > > > > > > On Fri, Oct 28, 2016 at 12:07 PM, Andrey Dyachkov < > > > andrey.dyach...@gmail.com > > > > wrote: > > > > > > > Hi Kafka developers :) > > > > > > > > We use Kafka for around half a year as a main backend system for > > storing > > > > events from big variety of internal microservices. In general It > works > > > > quite well and we experience issues only in some rare 'corner cases' > > > where > > > > we either run against a bug or do not understand why Kafka behaves in > > > > particular manner. The fact Kafka is the main system is used under > our > > > > software If it fails our system will fail too. That's why I am > writing > > to > > > > ask about the solution which can help us tolerate different issues on > > > Kafka > > > > side. First of all could you tell me why doesn't Kafka client have > the > > > > solution for this kind of problems? Have you ever had thoughts about > > it? > > > > Thank you in advance! > > > > -- > > > > > > > > With great enthusiasm, > > > > Andrey > > > > > > > > > > -- > > > > > > With great enthusiasm, > > > Andrey > > > > > > > > > > > -- > > Nacho (Ignacio) Solis > > Kafka > > nso...@linkedin.com > > > -- > > With great enthusiasm, > Andrey > -- Nacho (Ignacio) Solis Kafka nso...@linkedin.com
