ryerraguntla commented on code in PR #4205: URL: https://github.com/apache/iggy/pull/4205#discussion_r4031372725
########## gateways/kafka/docs/IDEMPOTENCE.md: ########## @@ -0,0 +1,92 @@ +# InitProducerId and idempotent producers + +Status: proposed. Answers the open half of +[#3545](https://github.com/apache/iggy/issues/3545) and gates the Phase 1 end-to-end test +([#3539](https://github.com/apache/iggy/issues/3539)), which drives +`kafka-console-producer.sh`. + +## The problem + +A stock Java producer sets `enable.idempotence=true` without being asked. That default arrived +in Kafka 3.0 and took effect from 3.0.1, 3.1.1 and 3.2.0, where a bug that suppressed it was +fixed. `kafka-console-producer.sh` leaves it on. + +An idempotent producer sends InitProducerId (key 22) before its first record. The gateway does +not list key 22, so ApiVersions does not advertise it, and the producer raises +`UnsupportedVersionException`. That exception is fatal. +`TransactionManager.maybeTransitionToErrorState` tests it above the `isTransactional()` branch. +The producer therefore enters a fatal error state instead of dropping back to weaker semantics. +It fails at startup, before it sends a record. + +The gateway's stated purpose is that a Kafka user swaps the broker and changes no application +code. A broker that the default producer cannot start against does not meet it. + +## Options + +| Option | Cost | What a stock producer does | +| -------- | ------ | ---------------------------- | +| Stub with `UNSUPPORTED_VERSION` | none | fails at startup unless the user sets `enable.idempotence=false` | +| Allocate only | about a day | works untouched, at-least-once delivery | +| Real deduplication | large | works untouched, exactly once per partition, single gateway instance only | + +Real deduplication means tracking a sequence number per producer and per partition, and +rejecting a duplicate or a gap. It is correct only while one gateway instance sees every write +from a producer, so it cannot be decided before the multi-instance question is. + +## Decision + +Allocate only. + +Rejecting the stock producer to avoid implementing deduplication trades the one requirement the +maintainers named against a guarantee Iggy does not offer today anyway. Allocating costs about +a day and keeps delivery exactly where it already is. + +## Behavior + +Add key 22 to `SUPPORTED_RANGES` in `src/protocol/api.rs` and advertise it through ApiVersions. +Without both, the producer never sends the request. `kafka-protocol` 0.18 carries the schemas, +request v0 to v5 and response v0 to v6, flexible from v2. + +InitProducerId with no `transactional_id`: + +- allocate the next producer id, return it with epoch 0 and error code 0 +- draw ids from a counter seeded per gateway instance, so two instances never hand out the same + id. Nothing reads the id today. Seeding it now is what stops a later deduplication layer from + being born broken + +InitProducerId with a `transactional_id`: + +- answer `UNSUPPORTED_VERSION` (35), unchanged. Transactions stay out of scope, and so do Review Comment: Should it be 35 or Use a feature error (INVALID_REQUEST / transactions-not-supported)? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
