[
https://issues.apache.org/jira/browse/KAFKA-5510?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16503878#comment-16503878
]
Guozhang Wang commented on KAFKA-5510:
--------------------------------------
Here is an idea for the general solution of this which is supposed to fix it
and the related issues listed above:
1. Remove the flag `commitOffsetNeeded` field in `StreamTask` class.
2. We keep track of the topic partitions whose consumed offsets has changed
since last commit, and then in `StreamTask#consumedOffsets`, depend on EOS
turned on or not:
2.1 If EOS turned off: we can filter out topic partitions that have changes
from last commit; if the resulted map is empty, skip the commit;
2.2 If EOS turned on: we send the empty map of commits even if it is empty.
> Streams should commit all offsets regularly
> -------------------------------------------
>
> Key: KAFKA-5510
> URL: https://issues.apache.org/jira/browse/KAFKA-5510
> Project: Kafka
> Issue Type: Bug
> Components: streams
> Reporter: Matthias J. Sax
> Priority: Major
>
> Currently, Streams commits only offsets of partitions it did process records
> for. Thus, if a partition does not have any data for longer then
> {{offsets.retention.minutes}} (default 1 day) the latest committed offset
> get's lost. On failure or restart {{auto.offset.rese}} kicks in potentially
> resulting in reprocessing old data.
> Thus, Streams should commit _all_ offset on a regular basis. Not sure what
> the overhead of a commit is -- if it's too expensive to commit all offsets on
> regular commit, we could also have a second config that specifies an
> "commit.all.interval".
> This relates to https://issues.apache.org/jira/browse/KAFKA-3806, so we
> should sync to get a solid overall solution.
> At the same time, it might be better to change the semantics of
> {{offsets.retention.minutes}} in the first place. It might be better to apply
> this setting only if the consumer group is completely dead (and not on "last
> commit" and "per partition" basis). Thus, this JIRA would be a workaround fix
> if core cannot be changed quickly enough.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
