Hi, If I understand correctly, the idempotent producer should be able to guarantee that messages are not duplicated when the producer retries sends, and as of https://issues.apache.org/jira/browse/KAFKA-5494, it should allow for the producer to have multiple in flight requests at a time without risking that the messages are reordered because batches are retried out of order.
The behavior I'm looking for is that if thread 1 sends message x=1, and thread 2 sees message x=1 come through Kafka, thread 2 can't then write message x=2 and have that be clobbered because thread 1's producer is retrying the send since it didn't receive the ack in time. The KafkaProducer Javadoc explains that when using the idempotent producer, "it is possible to continue sending after receiving an OutOfOrderSequenceException, but doing so can result in out of order delivery of pending messages. To ensure proper ordering, you should close the producer and create a new instance." I don't understand why this is needed? If I understand the feature correctly (please correct me), messages get sequence numbers, and requests fail if the messages are not received in order by the broker. Most out of order errors are handled internally by the producer, either by completing the request or by retrying. The case it can't handle is if the first pending message has a gap to the last acked sequence number, e.g. if the last acked number is 10 and the first message the producer has pending is 12. In that case there has been message loss, and the exception escapes from the producer to the caller. Is this the case the Javadoc note is referring to? Why is it necessary/helpful to terminate and replace the producer in order to guarantee ordering when this kind of gap occurs?
