We are in the middle of moving to 0.8.2 and migrating consumer offsets from Zookeeper to Kafka is a major reason for the move.
After upgrading, we see a new topic being created: Topic:__consumer_offsets PartitionCount:50 ReplicationFactor:3 Configs:segment.bytes=xxx,cleanup.policy=compact After passing a couple of messages through kafka mirror, I noticed that some of the partitions are changed: __consumer_offsets-18: total 0 -rw-r--r--. 1 kafka kafka xxx Nov 9 21:46 00000000000000000000.index -rw-r--r--. 1 kafka kafka 0 Nov 9 21:46 00000000000000000000.log __consumer_offsets-19: total 1624 -rw-r--r--. 1 kafka kafka xxx Nov 10 01:14 00000000000000000000.index -rw-r--r--. 1 kafka kafka 1652880 Nov 10 01:14 00000000000000000000.log However, /usr/local/kafka/bin/kafka-console-consumer.sh --zookeeper zkhost --topic __consumer_offts --from-beginning yields no result at all. Is it true that the offsets in special topic cannot be read directly like when they were in Zookeeper? We do have something that depends on direct access to consumer offsets. In addition, is it possible to reset the offsets in case of a mirror rewind. With Zookeeper, it is pretty easy to rewind the consumer offsets. I am still able to check the mirror lags via kafka.tools.ConsumerOffsetChecker. Thanks, Ye
