[
https://issues.apache.org/jira/browse/KAFKA-12776?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17344958#comment-17344958
]
NEERAJ VAIDYA commented on KAFKA-12776:
---------------------------------------
[~ableegoldman], [~ijuma], [~tombentley]
I think I kind of know why this issue occurs. Here is my take:
This issue or ordering mainly happens when there are multiple requests which
are being handled in parallel by the KafkaProducer and when metadata is being
fetched at the time of processing the request.
For example, consider a web service which accepts HTTP requests on the upstream
side. For each request (R1,R2,R3,Rn), if a KafkaProducer#send is invoked, and
if the Kafka cluster is down AND metadata is not available, then this will
cause each of R1,....Rn to be blocked in a metadata fetch call.
When the cluster is brought back online (within max.block.ms), then the first
request which succeeds in getting the metadata will successfully get pushed
onto the Producer buffer and eventually to the Topic.
The only way to control ordering when multiple parallel requests which are
stuck in a Metadata fetch would be to ensure that they are all serviced by a
single thread/queue ? I would have imagined that this must already be
happening, but apparently not.
> Producer sends messages out-of-order inspite of enabling idempotence
> --------------------------------------------------------------------
>
> Key: KAFKA-12776
> URL: https://issues.apache.org/jira/browse/KAFKA-12776
> Project: Kafka
> Issue Type: Bug
> Components: producer
> Affects Versions: 2.6.0, 2.7.0
> Environment: Linux RHEL 7.9 and Ubuntu 20.04
> Reporter: NEERAJ VAIDYA
> Priority: Major
> Attachments: mocker.zip
>
>
> I have an Apache Kafka 2.6 Producer which writes to topic-A (TA).
> My application is basically a Spring boot web-application which accepts JSON
> payloads via HTTP and then pushes each to a Kafka topic. I also use Spring
> Cloud Stream Kafka in the application to create and use a Producer.
> For one of my failure handling test cases, I shutdown the Kafka cluster while
> my applications are running. (Note : No messages have been published to the
> Kafka cluster before I stop the cluster)
> When the producer application tries to write messages to TA, it cannot
> because the cluster is down and hence (I assume) buffers the messages. Let's
> say it receives 4 messages m1,m2,m3,m4 in increasing time order. (i.e. m1 is
> first and m4 is last).
> When I bring the Kafka cluster back online, the producer sends the buffered
> messages to the topic, but they are not in order. I receive for example, m2
> then m3 then m1 and then m4.
> Why is that ? Is it because the buffering in the producer is multi-threaded
> with each producing to the topic at the same time ?
> My project code is attached herewith.
> I can confirm that I have enabled idempotence. I have also tried with
> ```max.in.flight.requests=1```
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
