Hi Michal, In your case you could try to increase the zookeeper session timeout value on the consumer side (default is 6 sec) and see if this is sufficient to cover the latency jitters.
Guozhang On Fri, Jul 11, 2014 at 5:25 AM, Michal Michalski < michal.michal...@boxever.com> wrote: > Hey Guozhang, > > Thanks for reply. I get your point on "hiding" some issues, but I'd prefer > to separate the recovery and reporting a failure. Also, I think if simple > restart is a possible solution, it shouldn't require implementing it > separately or, what's even worse, a manual intervention. Maybe I'll > describe my problem then to show you my point of view: > > ZK latency spiked for few seconds making ZK effectively dead from > consumers' point of view. Then they all reconnected. As I understand, when > it happened, it caused rebalancing. Some consumer groups succeeded, but > then another spike in latency happened and - as we suspect - it caused > rebalancing to fail, because creation of that ZK node failed at some point. > Ideally, I'd like to get notified about that problem (rebalancing failed > after X retries etc.), so I know there is an issue and I can investigate > it, but then I'd like Kafka consumer (or my app) to fallback to restart, > which could *possibly* make consumer recover. If not - that's my problem > then ;-) > > In our case it was enough to restart the app to get consumer working again, > but - as we didn't know about that behaviour before and we weren't prepared > for it - it required manual intervention (on Friday night, which made it > even more painful ;> ) which, we believe, wasn't necessary in that case and > could have been handled automatically. > > M. > > > > Kind regards, > Michał Michalski, > michal.michal...@boxever.com > > > On 10 July 2014 23:43, Guozhang Wang <wangg...@gmail.com> wrote: > > > Hi Michal, > > > > The rebalance will only be triggered on consumer membership or > > topic/partition changes. Once triggered it will try to finish the > rebalance > > for at most rebalance.max.retries times, i.e. if it fails it will wait > for > > rebalance.backoff.ms, and then try again until number of retries > > exhausted. > > When it happens an exception will be thrown and the consumer may be > fallen > > to a bad state. > > > > Then reason we did not implement automatic restart upon rebalance > failures > > is that it may actually "hide" some issues in the systems that actually > > caused the rebalance failure. The general design is that if some > > exception/errors are not expected like the rebalance failures we will let > > it to possibly hault/kill the instance rather than automatically restart > > and let it go. > > > > Guozhang > > > > > > > > > > On Thu, Jul 10, 2014 at 2:24 AM, Michal Michalski < > > michal.michal...@boxever.com> wrote: > > > > > Hi, > > > > > > Just wondering - is there any reason why rebalance.max.retries is 4 by > > > default? Is there any good reason why I shouldn't expect my consumers > to > > > keep trying to rebalance for minutes (e.g. 30 retries every 6 seconds), > > > rather than seconds (4 retries every 2 seconds by default)? > > > > > > Also, if my consumer fails to rebalance because of NoNodeException > > > (org.apache.zookeeper.KeeperException$NoNodeException: KeeperErrorCode > = > > > NoNode for > /consumers/is-entity-modified-document-group/ids/<something>) > > > wouldn't that make sense to make Kafka restart it automatically once it > > > "uses" all the retries attempts? Or recreate the inexistent ZK node > > like, I > > > believe, it will happen on consumer restart? > > > > > > I'm asking because that kind of errors seem to be "recoverable" ones, > > but - > > > if I understand it correctly - with current design they require > > > implementing additional mechanisms or manual intervention. > > > > > > > > > Kind regards, > > > Michał > > > > > > > > > > > -- > > -- Guozhang > > > -- -- Guozhang
