[
https://issues.apache.org/jira/browse/KAFKA-5247?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16010959#comment-16010959
]
Apurva Mehta commented on KAFKA-5247:
-------------------------------------
Note that this is only an issue when we mix offset commits from the consumer
and from the transactional producer, which may conceivably be the case during
an upgrade. If we don't fix this it is possible for the offset commits to
change after compaction.
For instance, if a transactional offset commit has position 100 in the log, and
consumer offset commit has position 101. After the transaction is committed,
the offset commit at position 100 will be materialized. If the coordinator
changes, the logic stays the same and the offset at position 100 will continue
to be materialized. However, if compaction kicks in, the offset at position
101 will now be materialized. This bug is about fixing this inconsistency so
that the same offset commit is always materialized in a stable state.
> Consumer GroupCoordinator should continue to materialize committed offsets in
> offset order even for transactional offset commits
> --------------------------------------------------------------------------------------------------------------------------------
>
> Key: KAFKA-5247
> URL: https://issues.apache.org/jira/browse/KAFKA-5247
> Project: Kafka
> Issue Type: Bug
> Reporter: Apurva Mehta
> Labels: exactly-once
> Fix For: 0.11.0.0
>
>
> In the TxnOffsetCommit patch, we thought it was ok for the group coordinator
> to use "transaction order" semantics when updating the cache, but we weren't
> thinking about the log cleaner.
> The log cleaner uses offset order when cleaning which means that the key with
> the largest offset always wins. So if we use transaction order when
> dynamically updating the cache, we will get different results from when we're
> loading the cache (even if the loading logic also uses transaction order).
> The fix should be straightforward: we need to remember the offset in the
> offsets topic of the offset that we cache. Then we only update it if the new
> entry has a higher offset.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
