krishvishal opened a new pull request, #4392:
URL: https://github.com/apache/iggy/pull/4392
## Which issue does this PR address?
Relates to #3542
## Rationale
The Kafka gateway needs to keep a committed offset for every Kafka group,
topic and partition. The accepted design in
`gateways/kafka/docs/OFFSET_STORAGE.md` stores it as an Iggy consumer group
offset under `kafka.cg.<group>`. That kind does not fit
the gateway:
- The server lets only a member that owns the partition write a group
offset, and the gateway never joins the group.
- The server refuses an offset past the last message. Kafka commits the
next offset to read, which is one past the last message, and it can commit 0 on
an empty partition.
- A group offset holds back retention. A Kafka committed offset never does.
- An Iggy consumer that polls that key with `Next` skips one record per
partition, and only a naming rule stops it.
The gateway handlers and the update to that design doc come in a separate
PR.
## What changed?
Group offsets had to come from a member of an Iggy consumer group, inside
the partition's range, and they held back retention. A new consumer kind,
`ExternalGroup` (wire value 3, `Consumer::external_group` in the Rust SDK), now
stores offsets for a group that lives outside Iggy. The key is the numeric id
of an existing Iggy consumer group, so the gateway still creates that group but
never joins it. These offsets have their own key space, so they never share a
key with the group's own offsets.
For this kind the server:
- accepts a store or a delete for any existing group, with no membership
check
- stores the value as sent, with no range check and no clamp at boot
- refuses a poll with `FeatureUnavailable`
- leaves it out of the retention barrier
- deletes the offsets of a deleted group, through the same replicated
delete as group offsets
The server replicates these offsets, recovers them at boot from
`offsets/external_groups/`, carries them in partition state transfer, and
clears them on a purge. Boot creates that directory for a partition that an
older build made. The HTTP client
returns `FeatureUnavailable` for this kind, because the REST API cannot
name a consumer kind. The CLI accepts `--kind external-group`.
Some changes elsewhere follow from the new kind:
- A caught-up primary now serves a state transfer of an empty partition
with committed ops. A client can store and then delete an offset on a partition
that never took a message. Before, a backup behind the repair floor had no way
past those ops.
- A purge whose offset directory sync fails leaves its generation
unrecorded, so the reconciler runs it again. Boot never clamps an external
offset, so without this a crash can bring back a deleted one.
- A state transfer install syncs `offsets/` after the kind directories.
The commit walk syncs it once per walk, not once per kind.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
