ryerraguntla commented on code in PR #4205: URL: https://github.com/apache/iggy/pull/4205#discussion_r4031391734
########## gateways/kafka/docs/OFFSET_STORAGE.md: ########## @@ -0,0 +1,132 @@ +# Consumer offset storage + +Status: proposed. Answers [#3540](https://github.com/apache/iggy/issues/3540) and blocks +[#3542](https://github.com/apache/iggy/issues/3542), OffsetCommit and OffsetFetch. + +## Decision + +Store Kafka group offsets as Iggy consumer offsets, one key per partition, under a consumer +group whose name is derived from the Kafka group id. + +The issue lists three options. None of them is this one. + +| Option | Why not | +| -------- | --------- | +| A, an Iggy-backed `__consumer_offsets` topic | Rebuilds what Iggy already has. A compacted offset topic needs compaction, which Iggy does not have, so the gateway replays the whole topic at every startup | +| B, a SQLite file on the gateway host | A second durability story, a second backup story, and offsets that do not survive moving the gateway | +| C, in memory only | Fails the acceptance criterion in #3542, which is that offsets survive a restart | + +Iggy already stores a durable offset per consumer and per partition, replicated with the +partition itself. Using it costs one call per partition on commit and one on fetch. + +## The key + +One Iggy consumer offset per Kafka `(group, topic, partition)`. + +- consumer kind: `ConsumerKind::ConsumerGroup` +- consumer id: `Identifier::named("kafka.cg.<group>")` +- stream and topic: whatever `TopicMapping` resolves the Kafka topic to +- partition: the Kafka partition index, unchanged, because both sides number from 0 + +The gateway calls `create_consumer_group(stream, topic, "kafka.cg.<group>")` before the first +commit for a group on a topic. If the group does not resolve in metadata, the server rejects the +offset write. The group has to exist first. The gateway never joins the group. Offsets are +readable by any client, member or not. + +### Why the group kind and not a named consumer + +`ConsumerKind::Consumer` with a name looks simpler, because it needs no registration call. It is +not. The server hashes a named consumer id to a `u32` with `XxHash32` +(`core/server/src/dispatch/partition.rs:916`), and that hash is the offset key. Two different +group names can collide and silently share one offset. + +A consumer group name resolves through metadata to a monotonic id instead. No hash, no +collision, and `get_consumer_groups(stream, topic)` lists what exists. + +### Why the prefix + +`kafka.cg.` keeps a Kafka group called `orders` off the key that a native Iggy consumer group +called `orders` uses. Without it the two share an offset and each one moves the other. + +The prefix does not make the offsets safe to poll with. That is the next section. + +## What is stored + +The Kafka committed offset, verbatim, with no conversion. + +The two systems mean different things by the number. Kafka commits the next offset to read. +Iggy stores the last offset processed, and `PollingKind::Next` resumes at the stored value plus +one (`core/partitions/src/iggy_partition.rs:3835`). A Kafka offset stored in an Iggy key is +therefore one greater than Iggy's own convention for that key. + Review Comment: How is Kafka s OffsetCommit offset -1 (forget commit) mapped in Iggy? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
