[
https://issues.apache.org/jira/browse/KAFKA-4385?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15642072#comment-15642072
]
Jiangjie Qin commented on KAFKA-4385:
-------------------------------------
Thanks for the explanation. Sorry I misunderstood the issue. However, I am
still not sure if this is the right way to solve the problem. Regarding the
number of (TopicMetadataRequest) TMR sent, the metadata request should not be
sent back to back, we have a metadata retry backoff which defaults to 100ms.
This configuration is shared by all the request retries. Is bumping retry
backoff time up an option for you? If the concern is about the total blocking
time, can you reduce the max.block.ms?
In general, we are trying to avoid adding specific configurations that exposes
the internal details. So it would be good to see if we can solve the problem
without doing that. Another option here is to reuse the retries configuration
for metadata refresh in {{waitOnMetadata()}}. Given that TMR shares
retry.backoff.ms, it seems reasonable to share the retries here as well.
> producer is sending too many unnecessary meta data request if the meta data
> for a topic is not available and "auto.create.topics.enable" =false
> -----------------------------------------------------------------------------------------------------------------------------------------------
>
> Key: KAFKA-4385
> URL: https://issues.apache.org/jira/browse/KAFKA-4385
> Project: Kafka
> Issue Type: Bug
> Reporter: Jun Yao
>
> All current kafka-client producer implementation (<= 0.10.1.0),
> When sending a msg to a topic, it will first check if meta data for this
> topic is available or not,
> when not available, it will set "metadata.requestUpdate()" and wait for meta
> data from brokers,
> The thing is inside "org.apache.kafka.clients.Metadata.awaitUpdate()", it's
> already doing a "while (this.version <= lastVersion)" loop waiting for new
> version response,
> So the loop inside
> "org.apache.kafka.clients.producer.KafkaProducer.waitOnMetadata() is not
> needed,
> When "auto.create.topics.enable" is false, sending msgs to a non-exist topic
> will trigger too many meta requests, everytime a metadata response is
> returned, because it does not contain the metadata for the topic, it's going
> to try again until TimeoutException is thrown;
> This is a waste and sometimes causes too much overhead when unexpected msgs
> are arrived.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
