[
https://issues.apache.org/jira/browse/KAFKA-20845?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Sean Quah updated KAFKA-20845:
------------------------------
Description:
Introduced by KAFKA-19760 while attempting to fix writes for large compressible
records.
When downgrading a consumer group, we apply changes to the group coordinator
state directly and do not replay the produced records. However, when appending
a large but compressible batch, we can allocate an empty batch, then flush,
then allocate another empty batch. Flushing an empty batch silently fails the
empty batch and rolls back coordinator state. We then write the records for the
downgrade, which leaves the in-memory state inconsistent with the state on disk.
After further writes, the state on disk becomes unloadable because the next
consumer group records cannot be applied to the classic group.
# We should not flush an empty batch when appending large batches.
# We could consider succeeding empty batches instead.
NB: Both succeeding and failing empty batches leads to divergence between
in-memory state and disk state when there are write operations that update the
state directly without replay.
When succeeding empty batches, a write operation that updates state directly
and then fails to serialize its records could leave an empty batch. When the
batch is committed, we end up with in-memory changes without corresponding
records on disk.
When failing empty batches, a lingering empty batch followed by a write
operation that updates state directly and then flushes immediately will have
its in-memory changes revert whilst records are written to disk.
was:
When downgrading a consumer group, we apply changes to the group coordinator
state directly and do not replay the produced records. However, when appending
a large but compressible batch, we can allocate an empty batch, then flush,
then allocate another empty batch. Flushing an empty batch silently fails the
empty batch and rolls back coordinator state. We then write the records for the
downgrade, which leaves the in-memory state inconsistent with the state on disk.
After further writes, the state on disk becomes unloadable because the next
consumer group records cannot be applied to the classic group.
# We should not flush an empty batch when appending large batches.
# We could consider succeeding empty batches instead.
NB: Both succeeding and failing empty batches leads to divergence between
in-memory state and disk state when there are write operations that update the
state directly without replay.
When succeeding empty batches, a write operation that updates state directly
and then fails to serialize its records could leave an empty batch. When the
batch is committed, we end up with in-memory changes without corresponding
records on disk.
When failing empty batches, a lingering empty batch followed by a write
operation that updates state directly and then flushes immediately will have
its in-memory changes revert whilst records are written to disk.
> Consumer group downgrades can leave group in invalid state when classic group
> metadata is very large
> ----------------------------------------------------------------------------------------------------
>
> Key: KAFKA-20845
> URL: https://issues.apache.org/jira/browse/KAFKA-20845
> Project: Kafka
> Issue Type: Bug
> Components: group-coordinator
> Affects Versions: 4.2.0, 4.3.0, 4.0.2, 4.1.1
> Reporter: Sean Quah
> Assignee: Sean Quah
> Priority: Blocker
> Fix For: 4.4.0
>
>
> Introduced by KAFKA-19760 while attempting to fix writes for large
> compressible records.
> When downgrading a consumer group, we apply changes to the group coordinator
> state directly and do not replay the produced records. However, when
> appending a large but compressible batch, we can allocate an empty batch,
> then flush, then allocate another empty batch. Flushing an empty batch
> silently fails the empty batch and rolls back coordinator state. We then
> write the records for the downgrade, which leaves the in-memory state
> inconsistent with the state on disk.
> After further writes, the state on disk becomes unloadable because the next
> consumer group records cannot be applied to the classic group.
> # We should not flush an empty batch when appending large batches.
> # We could consider succeeding empty batches instead.
> NB: Both succeeding and failing empty batches leads to divergence between
> in-memory state and disk state when there are write operations that update
> the state directly without replay.
> When succeeding empty batches, a write operation that updates state directly
> and then fails to serialize its records could leave an empty batch. When the
> batch is committed, we end up with in-memory changes without corresponding
> records on disk.
> When failing empty batches, a lingering empty batch followed by a write
> operation that updates state directly and then flushes immediately will have
> its in-memory changes revert whilst records are written to disk.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
