krishvishal opened a new pull request, #4415: URL: https://github.com/apache/iggy/pull/4415
Closes #3542 With `IGGY_KAFKA_BRIDGE_ENABLED=true`, OffsetCommit (8) and OffsetFetch (9) now keep group offsets in Iggy, using the `ExternalGroup` kind from #4392. A group resumes from its last commit, also after a gateway restart, and `kafka-console-consumer.sh --group` commits and resumes. ### How offsets are stored - key: the partition, plus Iggy consumer group `kafka.cg.<group>` on the mapped topic. The gateway creates that group on the first commit and never joins it - value: the Kafka offset as sent, no range check. 0 on an empty partition and one past the last message both work - a negative offset deletes the key. Metadata and leader epoch are dropped, OffsetFetch returns `""` and -1 - nothing deletes these groups on its own. `OFFSET_STORAGE.md` has the `iggy consumer-group delete` command for cleanup ### Commit and fetch - OffsetCommit v2-v9 checks the committer the way Kafka's `validateOffsetCommit` does. After a gateway restart a member gets `UNKNOWN_MEMBER_ID` and rejoins. Admin tools commit with a negative generation, which works on an empty group - OffsetFetch v1-v9, batched groups from v8. A null topic list (`Admin.listConsumerGroupOffsets`) reads every topic that holds the group - no key, no group or no topic reads as -1 with no error. Any other failed read fails the whole group instead, because a consumer that gets -1 resets its position - retriable failures go back as 14. Java fails offset calls on 6, and OffsetFetch on 7 - anything sent as -1 is logged at `error!` with the group and the cause ### Iggy calls - offset calls have their own 4 Iggy clients and don't wait behind Produce - each partition sticks to one client, which keeps its calls in order even when a caller gives up - a request gives up after half the min session timeout (3s by default). Heartbeats on the same connection wait behind it. Partitions cut off by that answer 14 without a call ### Server change `get_consumer_offset` answered a read that timed out on the owning shard with an empty body. The SDK reads that as "no offset", and a consumer would reset. It now answers `TransientNotAccepted`, which the Rust SDK retries. This is in `core/server/src/dispatch/partition.rs` and applies to every client. ### Known limits - every Iggy server needs a build with kind 3 (#4392) before the first commit. Nothing enforces that yet, and in a partly upgraded cluster one commit can stop a partition from committing - in a cluster an offset read can land on a node that lags. Right after a group's first commit that node answers -1, and a consumer on `auto.offset.reset=latest` skips records. Reads from the primary are #4409 - delivery stays at-least-once -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
