[
https://issues.apache.org/jira/browse/KAFKA-1510?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14083237#comment-14083237
]
Joel Koshy commented on KAFKA-1510:
-----------------------------------
I see - my thinking was since we retry indefinitely until a successful commit
it only needs to be done at the end (once) after a successful commit to Kafka.
So to summarize - let me know if you have any comments/questions:
* Can you fix the issue you caught with the isAutoCommit flag?
* Probably unnecessary to have the "|| forceCommit"
* Also, as mentioned in the summary I think it is reasonable to switch the
default offsets.storage to Kafka and set dual.commit to true.
* Can you also run the unit tests and verify? It will be useful to also run the
system tests (at least the mirror maker test suite). See
https://cwiki.apache.org/confluence/display/KAFKA/Kafka+System+Tests#KafkaSystemTests-RunningSystemTest
for more information on this - it should be sufficient to just run the mirror
maker tests.
> Force offset commits when migrating consumer offsets from zookeeper to kafka
> ----------------------------------------------------------------------------
>
> Key: KAFKA-1510
> URL: https://issues.apache.org/jira/browse/KAFKA-1510
> Project: Kafka
> Issue Type: Bug
> Affects Versions: 0.8.2
> Reporter: Joel Koshy
> Assignee: Joel Koshy
> Labels: newbie
> Fix For: 0.8.2
>
> Attachments: forceCommitOnShutdownWhenDualCommit.patch
>
>
> When migrating consumer offsets from ZooKeeper to kafka, we have to turn on
> dual-commit (i.e., the consumers will commit offsets to both zookeeper and
> kafka) in addition to setting offsets.storage to kafka. However, when we
> commit offsets we only commit offsets if they have changed (since the last
> commit). For low-volume topics or for topics that receive data in bursts
> offsets may not move for a long period of time. Therefore we may want to
> force the commit (even if offsets have not changed) when migrating (i.e.,
> when dual-commit is enabled) - we can add a minimum interval threshold (say
> force commit after every 10 auto-commits) as well as on rebalance and
> shutdown.
> Also, I think it is safe to switch the default for offsets.storage from
> zookeeper to kafka and set the default to dual-commit (for people who have
> not migrated yet). We have deployed this to the largest consumers at linkedin
> and have not seen any issues so far (except for the migration caveat that
> this jira will resolve).
--
This message was sent by Atlassian JIRA
(v6.2#6252)
