If you use the high level consumer implementation, and register all consumers as part of the same group - they will load-balance automatically.
When you add a consumer to the group, if there are enough partitions in the topic, some of the partitions will be assigned to the new consumer. When a consumer crashes, once its node in ZK times out, other consumers will get its partitions. Gwen On Wed, Oct 8, 2014 at 10:39 AM, Sharninder <sharnin...@gmail.com> wrote: > Hi, > > I'm not even sure if this is a valid use-case, but I really wanted to run > it by you guys. How do I load balance my consumers? For example, if my > consumer machine is under load, I'd like to spin up another VM with another > consumer process to keep reading messages off any topic. On similar lines, > how do you guys handle consumer failures? Suppose one consumer process gets > an exception and crashes, is it possible for me to somehow make sure that > there is another process that is still reading the queue for me? > > -- > Sharninder
