On Thu, Feb 8, 2018 at 4:27 PM, Luke Steensen < luke.steen...@braintreepayments.com> wrote:
Offsets are maintained per consumer group. When an individual consumer > crashes, the consumer group coordinator will detect that failure and > trigger a rebalance. This redistributes the partitions being consumed > across the available consumer processes, using the most recently committed > offset for each as the starting point. > Excellent, the getting started guide uses "consumer" sometimes meaning an individual consumer, and sometimes meaning a consumer group. That difficults a bit understanding how it works with exactitude. Thanks for clarifying. Let me followup with these questions then: 1) The group coordinator runs in Kafka? Or is the client library responsible for that? 2) Say that a consumer group has consumers A, B and C, assigned to the 3 partitions resectively. Consumer A polls and gets messages 75-80, but when it is processing message 77 crashes. The coordinator rebalances and assigns that partition to some of the other two, but at which offset is that partition left? 3) If the answer is 81, a critical consumer group that cannot miss messages is expected to write custom coordination code to avoid missing 77-80? If yes, are there best practices out there for doing this?
