Hi, It would be short lived consumer sessions: setup consumer + consumer group, produce to worker, accept response, teardown consumer. So it would stay well beneath the maxopenfile threshold.
What I was worried about is the potential overhead of creating and tearing down consumer groups since it generates zookeeper interaction. - pyr On Thu, 29 Sep 2011 08:35:33 -0700 Jun Rao <jun...@gmail.com> wrote: > Pierres-Yves, > > Since each consumer maintains its own state, there is relatively > little overhead per consumer on the broker. However, each consumer > has to maintain a socket connection to a broker, you can't have > infinite number of consumers. How many concurrent consumer groups do > you plan to have? > > Thanks, > > Jun > > On Thu, Sep 29, 2011 at 7:44 AM, Pierre-Yves ritschard > <p...@smallrivers.com>wrote: > > > Hi, > > > > One of the standard use cases for messaging queues is to implement > > some sort of distributed RPC. > > > > Let's pretend you have a bunch of worker processes, all registered > > in the same consumer group (to make sure only one is done at a > > time) and which are registered on a queue named to represent the > > task they know how to do. > > > > RPC could then be implemented by a dynamic consumer group creation > > and registration on the client side, then sending a message > > containing the created id in the payload, so that the worker > > process knows who to respond to. > > > > What I'm worried about is the overhead of registering new consumer > > groups on the fly, would that put an anormal burden on the brokers ? > > > > - pyr > >
