Hi Weide The consumer rebalancing algorithm is deterministic. In your failure scenario, when A comes back up again, the consumer threads will rebalance. This will give you the initial consumer configuration at the start of the test.
I'm unsure whether the partitions are balanced round robin, or if they will all go to A, then the overflow to B. If all of the messages need to be processed by a single machine, an alternative architecture would be to have a standby server that waits until master A fails and then connects as a consumer. This could be accomplished by watching Zookeeper and getting a notification when A's ephemeral node is removed. The high level consumer does seem to be the way to go as long as your application can handle duplicate processing. Daniel. > On 2/08/2014, at 1:38 pm, Weide Zhang <weo...@gmail.com> wrote: > > Hi Guozhang, > > If I use high level consumer, how do I ensure all data goes to master even > if slave was up and running ? Is it just by forcing master to have enough > consumer thread to cover maximum number of partitions of a topic since > high level consumer doesn't have assumption of consumers who are master and > consumers who are slave. > > For example, master A initiate enough thread such that it can cover all the > partitions. slave B is standby with same consumer group and same number of > threads but since master A has enough thread to cover all the partitions. > Slave B won't get any data. > > Suddenly master A goes down, slave B becomes new master, and it start to > get data based on high level consumer rebalance design. > > After that old master A comes up and becomes slave, will A get data ? Or A > will not get data because B has enough thread to cover all partitions in > the rebalancing logic. > > Thanks, > > Weide > > >> On Fri, Aug 1, 2014 at 4:45 PM, Guozhang Wang <wangg...@gmail.com> wrote: >> >> Hello Weide, >> >> That should be doable via high-level consumer, you can take a look at this >> page: >> >> https://cwiki.apache.org/confluence/display/KAFKA/Consumer+Group+Example >> >> Guozhang >> >> >>> On Fri, Aug 1, 2014 at 3:20 PM, Weide Zhang <weo...@gmail.com> wrote: >>> >>> Hi, >>> >>> I have a use case for a master slave cluster where the logic inside >> master >>> need to consume data from kafka and publish some aggregated data to kafka >>> again. When master dies, slave need to take the latest committed offset >>> from master and continue consuming the data from kafka and doing the >> push. >>> >>> My questions is what will be easiest kafka consumer design for this >>> scenario to work ? I was thinking about using simpleconsumer and doing >>> manual consumer offset syncing between master and slave. That seems to >>> solve the problem but I was wondering if it can be achieved by using high >>> level consumer client ? >>> >>> Thanks, >>> >>> Weide >> >> >> >> -- >> -- Guozhang >>
