So lets say I have a topic with 100 partitions. And Im running a high-kevel consumer with 25 threads. If I start a new consumer process with the same groupid and say this has another 10 threads. I can expect some partitions to be reassigned to the new consumers right ?
On Jun 11, 2012, at 8:05 PM, Sam William wrote: > Thanks Jay.. I had made such an assumption and seems like it was valid. > > Sam > On Jun 11, 2012, at 7:52 PM, Jay Kreps wrote: > >> Hi Sam, >> >> We assign partitions to consumer threads so that each partition has a >> consumer. If you have more partitions then threads, some threads will get >> multiple partitions (nothing wrong with this); if you have more threads >> than partitions some thread will not get any partitions and will be idle. >> But never will one partition go to more than one thread (since the >> consumption is meant to be ordered within that partition, splitting it >> would likely break this). >> >> -Jay >> >> On Mon, Jun 11, 2012 at 7:11 PM, Sam William <sa...@stumbleupon.com> wrote: >> >>> Hi, >>> Im trying out the high level consumer using the ConsumerConnector API . >>> I have a topic with 100 partitions . What does it mean if I create >>> messagestreams with >>> >>> >>> consumerConnector.createMessageStreams(Predef.Map(topic -> partitions), >>> decoder) >>> >>> where partitions < 100 (say 50). does the consumer read messages only >>> from 50 of the 100 partitions ? >>> >>> >>> >>> >>> >>> Sam William >>> sa...@stumbleupon.com >>> >>> >>> >>> > > Sam William > sa...@stumbleupon.com > > > Sam William sa...@stumbleupon.com
