hechengbo created KAFKA-21516:
---------------------------------
Summary: Delayed InitializeShareGroupState retries can overwrite
persisted consumption progress
Key: KAFKA-21516
URL: https://issues.apache.org/jira/browse/KAFKA-21516
Project: Kafka
Issue Type: Bug
Affects Versions: 4.5.0
Reporter: hechengbo
h1. Problem
ShareCoordinatorShard.initializeState treats an initialization request as a
duplicate only when both stateEpoch and startOffset match the persisted state.
Normal consumption can advance startOffset without changing stateEpoch.
Consequently, an identical initialization retry arriving after consumption has
progressed is no longer recognized as a duplicate. It generates a new
initialization snapshot, overwriting the persisted start offset and clearing
acknowledgment batches and delivery completion count.
h1. Trigger scenario
When initializing newly added partitions of an already subscribed topic, the
Group Coordinator uses startOffset=0.
Timed-out initialization can be rescheduled while earlier persister
initialization attempts are still retrying. If these tasks overlap within the
same state epoch, one may complete and allow consumption to begin before
another initialization retry arrives.
The delayed retry can then overwrite consumption state already persisted by the
active share partition.
h1. Deterministic reproduction
A regression test in ShareCoordinatorShardTest replays every generated
coordinator record and performs the following sequence:
1. Initialize a share partition with stateEpoch=5 and startOffset=0.
2. Read its state with leaderEpoch=1.
3. Persist consumption progress with the same stateEpoch, startOffset=20, and
acknowledged offsets 25–29.
4. Retry the original initialization request: stateEpoch=5, startOffset=0.
5. Read the state again.
Expected:
The retry preserves startOffset=20 and the acknowledgment batches.
Actual:
startOffset becomes 0 and the acknowledgment batches become empty.
The attached reproduction-only patch adds:
testInitializeStateRetryPreservesConsumptionProgress
Run after applying the patch:
./gradlew :share-coordinator:test --tests
org.apache.kafka.coordinator.share.ShareCoordinatorShardTest.testInitializeStateRetryPreservesConsumptionProgress
-PmaxTestRetries=0 --console=plain
h1. Impact
The test confirms regression of the persisted share-partition start offset and
loss of persisted acknowledgment state.
Acknowledged records may be delivered again after the share-partition cache is
reconstructed, for example following a leader change or restart, if the records
are still retained and the lost state has not been restored. Immediate
duplicate delivery is not required to trigger this issue.
h1. Possible fix
Treat initialization as a no-op when state already exists at the requested
stateEpoch, regardless of startOffset. A higher stateEpoch would continue to
allow an explicit reset.
Current trunk increments the group epoch before an administrative share-group
offset reset.
Compatibility consideration: older Group Coordinator implementations, including
4.1.0, can perform legitimate resets without incrementing the epoch. The
proposed change would ignore such same-epoch resets during mixed-version
operation.
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)