Joel Koshy created KAFKA-1510:
---------------------------------
Summary: Force offset commits at a minimum interval when migrating
consumer offsets from zookeeper to kafka
Key: KAFKA-1510
URL: https://issues.apache.org/jira/browse/KAFKA-1510
Project: Kafka
Issue Type: Bug
Affects Versions: 0.8.2
Reporter: Joel Koshy
Fix For: 0.8.2
When migrating consumer offsets from ZooKeeper to kafka, we have to turn on
dual-commit (i.e., the consumers will commit offsets to both zookeeper and
kafka) in addition to setting offsets.storage to kafka. However, when we commit
offsets we only commit offsets if they have changed (since the last commit).
For low-volume topics or for topics that receive data in bursts offsets may not
move for a long period of time. Therefore we may want to force the commit (even
if offsets have not changed) when migrating (i.e., when dual-commit is enabled)
- we can add a minimum interval threshold (say force commit after every 10
auto-commits) as well as on rebalance and shutdown.
Also, I think it is safe to switch the default for offsets.storage from
zookeeper to kafka and set the default to dual-commit (for people who have not
migrated yet). We have deployed this to the largest consumers at linkedin and
have not seen any issues so far (except for the migration caveat that this jira
will resolve).
--
This message was sent by Atlassian JIRA
(v6.2#6252)
