Brandon Jiang created KAFKA-9158:
------------------------------------
Summary: producer fetch metadata until buffer is full
Key: KAFKA-9158
URL: https://issues.apache.org/jira/browse/KAFKA-9158
Project: Kafka
Issue Type: Improvement
Components: producer
Affects Versions: 2.2.1
Reporter: Brandon Jiang
Currently, based on my understanding with KafakProducer.doSend() method, when
trigger kafka java client with
{code:java}
org.apache.kafka.clients.producer.Producer.send(message);{code}
The currently behavior is to do metadata fetch first and then queue the message
for sending ? This could result the kafka producer hangs for max.block.ms time
when kafka servers are down.
Is it possible to change the Producer.send() function behavior to queue all the
messages first and then only do metadata fetch async when the queue is full.
e.g.
{code:java}
if (msgQueue.batchIsFull ) {
clusterAndWaitTime = waitOnMetadata(record.topic(), record.partition(),
maxBlockTimeMs);
.....
this.sender.wakeup();
}{code}
I am happy to make this changes. But before doing that, just want to check the
current design logic behind producer.send().
I have seen there are lots of talks regarding the producer send() behavior.
e.g. https://issues.apache.org/jira/browse/KAFKA-2948,
https://issues.apache.org/jira/browse/KAFKA-5369
So trying to make sure I didnot misunderstand the current behavior here.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
