Peter, We probably shouldn't commit offsets automatically during rebalance, if auto offset commit is turned off. The caveat is that during a rebalance, all message since the last manual commit point will be redelivered to the consumer. Could you create a jira and maybe submit a patch for this?
Thanks, Jun On Thu, May 17, 2012 at 8:12 AM, Peter Romianowski <honkb...@googlemail.com>wrote: > Hi, > > should have found a better subject... > > If I read the source correctly then offsets can be committed at any > time (whenever there is a change in consumer or broker zk registry). > Our application doesn't use auto-commit in order to batch some > messages together, process them and if everything went fine, we call > commitOffsets(). If, for any reason, the processing of messages does > not succeed, we rely on Kafka's promise to re-deliver the messages. > > But if ZKRebalancerListener triggers a rebalance before our "batch" of > messages is full, then offsets will be committed even if the messages > have not been processed yet by our application. So if then processing > of these messages fails, we basically lost them, right? > > Are there any recipes how to handle this? I really would like to stick > to ZookeeperConsumerConnector, so switching to SimpleConsumer would be > considered as "last resort". > > Regards > > Peter > > -- > 404 Signature Not Found >
