Hey Chinmay, I remember someone else having this issue with Kafka + Zookeeper. IIRC, the cause was ZkClient blocking indefinitely.
You may find this useful. https://issues.apache.org/jira/browse/KAFKA-1907 http://mail-archives.apache.org/mod_mbox/kafka-dev/201501.mbox/browser Aditya ________________________________________ From: Chinmay Soman [chinmay.cere...@gmail.com] Sent: Friday, February 20, 2015 10:15 AM To: dev@samza.apache.org Subject: Re: Question on hello-samza (Kafka startup and shutdown) I haven't really figured it out. But just to clarify - I'm not starting stopping within 5 seconds of each other - its more like a couple of hours. The Kafka process is indeed running even after stop all : It seems to be waiting on Zookeeper (doing a lot of retries). If I bring up Zookeeper again - then the Kafka process shuts down cleanly :) But yes - in most cases I'm using SIGKILL and not SIGTERM to resolve this. This is not really an urgent issue - but was just curious - what's really happening ? On Fri, Feb 20, 2015 at 8:47 AM, Chris Riccomini <criccom...@apache.org> wrote: > Hey Chinmay, > > It seems controlled.shutdown.enable=true is the default. Chinmay, did you > figure this out? I haven't seen this before, but I don't usually stop/start > within 5s of eachother. > > One thing that you might have a look at is whether the Kafka or ZK > processes are living past bin/grid stop all. I have seen procs (NM and > Kafka usually) continue to be alive after `stop all` is executed. I think > this is because the stop scripts SIGTERM and return immediately. This > allows procs to do a cleaner shutdown. But if you stop/start quickly, you > might get some weirdness there. Try jps'ing in between the stop/start, and > check to make sure there's nothing still alive (wait in a loop until > everything shuts down cleanly, and kill -9 if it takes more than 60s, or > something). > > Cheers, > Chris > > On Thu, Feb 19, 2015 at 2:01 PM, Neha Narkhede <neha.narkh...@gmail.com> > wrote: > > > Depending on the version of Kafka you're at, "controlled.shutdown.enable" > > should be set to true. If that's true and you always shutdown the broker > > cleanly (kill -15, not kill -9) and there are more than 1 replicas > > available, you should not see LeaderNotAvailable exceptions. If you kill > > the broker (kill -9) then Kafka does not get a chance to move the leaders > > away from the broken being shut down and the leader re-election can take > > some time leading to many LeaderNotAvailable exceptions. > > > > You can verify the replica availability as well as leader movement > through > > the kafka-topics command before shutting down zookeeper. > > > > Thanks > > Neha > > > > On Thu, Feb 19, 2015 at 10:51 AM, Felix GV > <fville...@linkedin.com.invalid > > > > > wrote: > > > > > I'm not 100% sure, but I think this happens when ZK ephemeral znodes > have > > > not had time to expire properly. When Kafka shuts down gracefully, it > > > should clean up its ephemeral nodes immediately (presumably, but that > is > > > also an assumption... maybe it does have a short-coming in its graceful > > > shutdown logic). If Kafka gets killed improperly and bounced back up > > right > > > away, it cannot assume leadership properly because the ephemeral znodes > > of > > > the previous run are still there in ZK. > > > > > > I imagine Kafka could have some logic to deal with that better when it > > > gets fast-bounced... Alternatively, you may just have to wait a bit > > before > > > restarting Kafka after killing it. > > > > > > If anyone knows better, please correct me if I'm wrong. > > > > > > -- > > > > > > Felix GV > > > Data Infrastructure Engineer > > > Distributed Data Systems > > > LinkedIn > > > > > > f...@linkedin.com > > > linkedin.com/in/felixgv > > > > > > ________________________________________ > > > From: Chinmay Soman [chinmay.cere...@gmail.com] > > > Sent: Thursday, February 19, 2015 10:44 AM > > > To: dev@samza.apache.org > > > Subject: Question on hello-samza (Kafka startup and shutdown) > > > > > > Sending to a wider audience to know if anyone is also seeing this > issue. > > > > > > It seems Kafka gets in a weird state everytime I do bin/grid stop all > > (and > > > then start all). > > > > > > I keep getting a LeaderNotAvailable exception on the producer side. It > > > seems this happens everytime Kafka hasn't been shut down properly. This > > > issue goes away if I use the following sequence: > > > > > > * bin/grid stop kafka > > > * bin/grid stop zookeeper (after like 5 seconds). > > > > > > (and then start everything). > > > > > > Has anyone else seen this ? > > > > > > -- > > > Thanks and regards > > > > > > Chinmay Soman > > > > > > -- Thanks and regards Chinmay Soman
