Hi Andrew. Is this what you're saying: - sometimes you get stuck (like a blocking call) when you call some function in the kafka client - you can go around this case (by wrapping it) such that if the kafka client call gets stuck your software doesn't get stuck - you're wondering why the kafka client is not more resilient to the broker being unavailable/down
Is this correct? Nacho On Mon, Oct 31, 2016 at 1:54 PM, Andrey Dyachkov <andrey.dyach...@gmail.com> wrote: > Hi Becket, > > I asked the same question in the user mailing list and we had a small > discussion there. > > We have default timeout for producer and consumer it is 30 secs. > If Kafka is not available we will wait 30 secs on each request. > Decreasing timeout won't help much because we have a lot requests > but circuiting them will help us tremendous. We could implement such > logic before actually calling Kafka client but there is problem what the > trigger is to open the circuit. I thought about circuiting by unavailable > broker > but it is quite an expansive operation to figure out in the client's code > on the flip side Kaka client already does all the job to make out the > broker. > > For us very important to stay live even if Kafka is unavailable (it > happens) that's why I think avoiding cascading issues will be very useful. > > The circuit breaker here is a thing which can stop requests to Kafka for > certain period of time until a specific topic/partition/broker is not > available. > > On Mon, 31 Oct 2016 at 21:38 Becket Qin <becket....@gmail.com> wrote: > > Hi Andrey, > > I am not sure I understand the question "why doesn't Kafka client have > the solution > for this kind of problems?" Can you elaborate a little on what kind of > problems are you referring to? And what do you mean by "circuit breaker" > here? > > Jiangjie (Becket) Qin > > On Fri, Oct 28, 2016 at 12:07 PM, Andrey Dyachkov < > andrey.dyach...@gmail.com > > wrote: > > > Hi Kafka developers :) > > > > We use Kafka for around half a year as a main backend system for storing > > events from big variety of internal microservices. In general It works > > quite well and we experience issues only in some rare 'corner cases' > where > > we either run against a bug or do not understand why Kafka behaves in > > particular manner. The fact Kafka is the main system is used under our > > software If it fails our system will fail too. That's why I am writing to > > ask about the solution which can help us tolerate different issues on > Kafka > > side. First of all could you tell me why doesn't Kafka client have the > > solution for this kind of problems? Have you ever had thoughts about it? > > Thank you in advance! > > -- > > > > With great enthusiasm, > > Andrey > > > > -- > > With great enthusiasm, > Andrey > -- Nacho (Ignacio) Solis Kafka nso...@linkedin.com
