Jeff, Your understanding is correct. Operational wise, we have some jmx that gives consumer stats per topic. There is also a tool CheckOffsetLag that tells you how far behind a consumer is. For coordination btw producers and consumers, if you just want each message to be consumed by a consumer, not a particular one, there is no coordination needed.
Thanks, Jun On Tue, Jun 12, 2012 at 9:58 AM, Rodenburg, Jeff <jeff.rodenb...@teamaol.com > wrote: > Hi all - > > Just getting familiar with Kafka, and learning about consumer groups. > Hoping someone can provide some context here. > > As I understand it, consumers register with the broker and consume a > topic. Multiple consumers can consume a single topic, as a consumer group. > Each consumer actually gets a partition of messages, so there is no overlap > -- a single consumer within a group will receive a message on its > topic/partition. Consumer rebalancing is the process whereby members of a > consumer group are added and/or dropped from the group, and partitions are > sorted/reassigned to the current consumer group members. > > Some questions: > > * Is this accurate? What am I missing? > * Operationally, is consumer "failover" basically service monitoring at > the consumer process level? > * How much coordination is required between producers and consumers > around partitioning? (Automated, configuration, etc.) > * How are topics monitored for SLA on throughput/load, i.e. spinning up > consumers as needed for topic message spikes? > > Appreciate any further information and/or context anyone can share. > > cheers, > Jeff >
