I think I understand. It is not possible for two consumers in the same
consumer group to consume from the same partition therefore duplication
is not an issue.
Once again my confusion stems from the fact that I've been mostly
looking into the Ruby consumer/producers. It seems with the Ruby client
consumer groups are not available and hence only the pub-sub use case is
available. There also appears to be some other missing functionality
which further adds to my overall confusion.
On 11/11/11 7:17 PM, Jay Kreps wrote:
Inder is correct. To elaborate:
Each message is delivered to one consumer in each consumer group.
Traditional pub/sub delivers each message to each consumer. Traditional
queuing delivers each message to exactly one consumer. This model
generalizes both of those and anything in between--if you assign all
consumers the same consumer group it acts like a queue, if you assign them
all different consumer groups it acts like pub/sub.
In my experience you rarely want either traditional queues or traditional
pub/sub in internet-style systems. Each consumer usually MUST load balance
the consumption over multiple machines for throughput and failover but you
often do have multiple of these consuming clusters. This is what we are
attempting to capture.
Let me know if you think we could explain this better, we have really
struggled to make this clear to people I think.
-Jay
On Fri, Nov 11, 2011 at 6:30 PM, Mark<static.void....@gmail.com> wrote:
How do consumer groups work? Say I have 3 consumers each reading from
topic1 how do you ensue each consumer is not consuming a message that has
been consumed by another consumer?
