@Pradeep - I just read your thread, the 1hr pause was when all the consumers where shutdown simultaneously. I'm testing out rolling restart to get the actual numbers. The initial numbers are promising.
`STOP (1) (1min later kicks off) -> REBALANCE -> START (1) -> REBALANCE (takes 1min to get a partition)` In your thread, Ewen says - "The LeaveGroupRequest is only sent on a graceful shutdown. If a consumer knows it is going to shutdown, it is good to proactively make sure the group knows it needs to rebalance work because some of the partitions that were handled by the consumer need to be handled by some other group members." So does this mean that the consumer should inform the group ahead of time before it goes down? Currently, I just shutdown the process. On Fri, Feb 10, 2017 at 8:35 AM, Pradeep Gollakota <pradeep...@gmail.com> wrote: > I asked a similar question a while ago. There doesn't appear to be a way to > not triggering the rebalance. But I'm not sure why it would be taking > 1hr > in your case. For us it was pretty fast. > > https://www.mail-archive.com/users@kafka.apache.org/msg23925.html > > > > On Fri, Feb 10, 2017 at 4:28 AM, Krzysztof Lesniewski, Nexiot AG < > krzysztof.lesniew...@nexiot.ch> wrote: > > > Would be great to get some input on it. > > > > - Krzysztof Lesniewski > > > > > > On 06.02.2017 08:27, Praveen wrote: > > > >> I have a 16 broker kafka cluster. There is a topic with 32 partitions > >> containing real time data and on the other side, I have 32 boxes w/ 1 > >> consumer reading from these partitions. > >> > >> Today our deployment strategy is stop, deploy and start the processes on > >> all the 32 consumers. This triggers re-balancing and takes a long period > >> of > >> time (> 1hr). Such a long pause isn't good for real time processing. > >> > >> I was thinking of rolling deploy but I think that will still cause > >> re-balancing b/c we will still have consumers go down and come up. > >> > >> How do you deploy to consumers without triggering re-balancing (or > >> triggering one that doesn't affect your SLA) when doing real time > >> processing? > >> > >> Thanks, > >> Praveen > >> > >> > > >
