David Jacot created KAFKA-21450:
-----------------------------------
Summary: Iterating a TimelineHashMap at a snapshot returns
duplicate or missing entries after a revert
Key: KAFKA-21450
URL: https://issues.apache.org/jira/browse/KAFKA-21450
Project: Kafka
Issue Type: Bug
Components: group-coordinator, kraft
Affects Versions: 2.8.0
Reporter: David Jacot
Assignee: David Jacot
After SnapshotRegistry#revertToSnapshot(S), iterating a TimelineHashMap or a
TimelineHashSet at S, or at an older snapshot (entrySet(epoch), values(epoch),
keySet(epoch), iterator(epoch)), can return an entry twice or skip one, until S
is deleted. Lookups by key (get(key, epoch)), size(epoch) and reads at the
latest epoch are not affected.
Minimal reproduction:
{code:java}
SnapshotRegistry registry = new SnapshotRegistry(new LogContext());
TimelineHashMap<String, Integer> map = new TimelineHashMap<>(registry, 1);
map.put("a", 1);
registry.getOrCreateSnapshot(0);
map.put("a", 2);
registry.revertToSnapshot(0);
map.entrySet(0); // returns [a=1, a=1]
{code}
Cause: revertToSnapshot(S) copies the elements of S's tier back into the
current table in SnapshottableHashTable#executeRevert but leaves them in the
tier. The historical iterator at an epoch <= S reads both the current table and
the tiers, so every restored element is returned twice. If a restored element
is then removed or replaced again while S is still the latest snapshot, it goes
back into the same tier. The tier's table can then grow past the current table,
and the iterator skips slots, so entries are missed as well.
Effect: the group coordinator reverts to the last written offset when a write
fails, and serves reads at the last committed offset. Until the next write is
committed, ConsumerGroupDescribe, ShareGroupDescribe, StreamsGroupDescribe,
OffsetFetch for all the partitions of a group, and ListGroups can return a
member, a partition or a group twice, or miss one. In the KRaft controller, the
reads at the last stable offset collect most results into maps.
ListPartitionReassignments without topics could list a topic twice if a
controller that has just resigned answers before its next commit.
Fix: once the elements of S's tier are back in the current table, drop them
from the tier.
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)