jeffkbkim opened a new pull request, #14849:
URL: https://github.com/apache/kafka/pull/14849
After the new coordinator loads a __consumer_offsets partition, it logs the
following exception when making a read operation (fetch/list groups, etc):
```
java.lang.RuntimeException: No in-memory snapshot for epoch 740745. Snapshot
epochs are:
at
org.apache.kafka.timeline.SnapshotRegistry.getSnapshot(SnapshotRegistry.java:178)
at
org.apache.kafka.timeline.SnapshottableHashTable.snapshottableIterator(SnapshottableHashTable.java:407)
at
org.apache.kafka.timeline.TimelineHashMap$ValueIterator.<init>(TimelineHashMap.java:283)
at
org.apache.kafka.timeline.TimelineHashMap$Values.iterator(TimelineHashMap.java:271)
```
This happens because we don't have a snapshot at the last updated high
watermark after loading. We cannot generate a snapshot at the high watermark
after loading all batches because it may contain records that have not yet been
committed. We also don't know where the high watermark will advance up to so we
need to generate a snapshot for each offset the loader observes to be greater
than the current high watermark. Then once we add the high watermark listener
and update the high watermark we can delete all of the older snapshots.
### Committer Checklist (excluded from commit message)
- [ ] Verify design and implementation
- [ ] Verify test coverage and CI build status
- [ ] Verify documentation (including upgrade notes)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscr...@kafka.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
