krishvishal opened a new issue, #4513: URL: https://github.com/apache/iggy/issues/4513
Follow-up to #3545. ## Summary Produce ignores producer id, epoch and sequence (`gateways/kafka/docs/IDEMPOTENCE.md`). The gateway never answers `OUT_OF_ORDER_SEQUENCE_NUMBER` (45), `DUPLICATE_SEQUENCE_NUMBER` (46) or `UNKNOWN_PRODUCER_ID` (59). Who hits it: - The Java producer is idempotent by default since 3.0, with `acks=all`, 5 requests in flight and unlimited retries. franz-go and kafka-python 3.0 are idempotent by default too. - A batch whose response is lost is written twice. A batch retried while later batches are in flight lands after them. The Java client relies on the broker's 45 to keep the order. - The gateway sends retriable codes in normal operation. A send past its deadline gets `REQUEST_TIMED_OUT` (7), and a request past its budget gets `NOT_LEADER_OR_FOLLOWER` (6). A pipelined producer then reorders the records of one key, and nothing reports it. ## Depends on - #4502, one Iggy session per producer - The two SDK seams in `IDEMPOTENCE.md`: a client id chosen by the caller at build time, and a request id supplied by the caller ## Scope - [ ] Producer state per producer id and partition: the last five batch sequences and their base offsets, as Kafka's `ProducerStateManager` keeps them - [ ] A retry of a stored batch gets its first offset back. A gap gets 45 - [ ] The two SDK seams, so a send that timed out between the gateway and Iggy is retried as the same Iggy request - [ ] Restart safety: persist the state, or bump the epoch and answer 59 after a restart - [ ] Update `IDEMPOTENCE.md` and the "Delivery guarantees" section of `gateways/kafka/README.md` ## Acceptance criteria - [ ] Test: a Produce retried after a lost response writes each record once - [ ] Test: a retried batch with later batches in flight keeps the order of one key, or fails with 45 - [ ] E2E with the Java producer at default settings under injected timeouts: no duplicates and no reordering ## References - Producer defaults: [ProducerConfig.java:429](https://github.com/apache/kafka/blob/b15258c7398889248d7fbf2a2218cc01346ebf92/clients/src/main/java/org/apache/kafka/clients/producer/ProducerConfig.java#L429), [ProducerConfig.java:520-525](https://github.com/apache/kafka/blob/b15258c7398889248d7fbf2a2218cc01346ebf92/clients/src/main/java/org/apache/kafka/clients/producer/ProducerConfig.java#L520-L525), [ProducerConfig.java:574-578](https://github.com/apache/kafka/blob/b15258c7398889248d7fbf2a2218cc01346ebf92/clients/src/main/java/org/apache/kafka/clients/producer/ProducerConfig.java#L574-L578) - Order on retry: [TransactionManager.java:1089-1105](https://github.com/apache/kafka/blob/b15258c7398889248d7fbf2a2218cc01346ebf92/clients/src/main/java/org/apache/kafka/clients/producer/internals/TransactionManager.java#L1089-L1105) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
