I think Onur has once mentioned similar issues that currently we do not have a fully automated smooth upgrade path from old consumer (offsets stored in ZK) to new consumer (offsets stored in Kafka), maybe he can provide more context here and how it can be resolved.
Currently the following procedure would be a not-perfect mechanism I can think of: 1. Upgrade to 0.8.2 first, turn on dual.commit.enabled and offset.storage to zookeeper, ran for a while until you can make sure that at least one set of offsets has been written to brokers. 2. Stop all consumers, and the restart them with the newer version. Note that you cannot do rolling bounce to keep the application live during this upgrade since during the time there are both old and new consumers, since they will be coordinated differently reading offsets from different places (ZK v.s. brokers), they could potentially fetching the same partitions and hence cause duplicate data. Guozhang On Sun, Nov 13, 2016 at 5:45 PM, Wei Li <weili...@gmail.com> wrote: > Hi All, > > We recently upgraded kafka from 0.8 to 0.10 with security features enabled > (ssl/sasl/acls). One thing noticed is the offset is not longer kept on > zookeeper any more in 0.10 release. Though this is new feature for new > version, we like to still keep the offsets stored on zookeeper in short > terms without breaking the scripts/monitoring utilities that use old style > offset. By following the documentation, I have enabled > dual.commit.enabled=true and offset.storage=zookeeper. After rolling > restarted, the offsets are still not saved in zookeeper. There is no > errors recorded in server or zookeeper logs. Can someone suggest how I > can further debug this issue? BTW, I am using logstash as kafka client to > read data from kafka topic. > > Thanks, > Wei > -- -- Guozhang
