[
https://issues.apache.org/jira/browse/KAFKA-4385?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15642883#comment-15642883
]
Jun Yao commented on KAFKA-4385:
--------------------------------
yeah,
the case is
1. broker has disabled topic auto creation,
2. producer sending msg to a topic that does not exist on broker
so for 'max.block.ms', it's shared in multiple places, such as buffer is full,
and there are cases that the buffer is full, that I am fine to let caller of
producer.send() wait and be blocked so that upstream can be slowed down to wait
for producers to actually send out the data and make the buffer available.
while for the case in discussion, it's more that once the 1st meta data
response does not contain the topic metadata, there is no need to keep looping
the meta data, as that would be just a waste.
just skip this failure and move on for next record would be ideal from my
perspective. tuning "retry.backoff.ms" is just let it wait there for nothing...
for deprecating "metadata.fetch.timeout.ms" to use one config of
"max.block.ms", yeah, this might be more consistent.
for whether the case is valid, yeah, ideally the topics should be maintained
in a central place and the producers should do filtering on the messages based
on the config.
In reality it would still be possible to disable topic auto creation while the
producer may still send some minor invalid messages to brokers,
in such case the msgs will anyway be a failure that msgs will not be delivered.
But sending those duplicate metadata request or slow down other msgs is kind
of unnecessary. if the failure is just skip this record once, then from user's
perspective it's very elegant.
> producer is sending too many unnecessary meta data request if the meta data
> for a topic is not available and "auto.create.topics.enable" =false
> -----------------------------------------------------------------------------------------------------------------------------------------------
>
> Key: KAFKA-4385
> URL: https://issues.apache.org/jira/browse/KAFKA-4385
> Project: Kafka
> Issue Type: Bug
> Reporter: Jun Yao
>
> All current kafka-client producer implementation (<= 0.10.1.0),
> When sending a msg to a topic, it will first check if meta data for this
> topic is available or not,
> when not available, it will set "metadata.requestUpdate()" and wait for meta
> data from brokers,
> The thing is inside "org.apache.kafka.clients.Metadata.awaitUpdate()", it's
> already doing a "while (this.version <= lastVersion)" loop waiting for new
> version response,
> So the loop inside
> "org.apache.kafka.clients.producer.KafkaProducer.waitOnMetadata() is not
> needed,
> When "auto.create.topics.enable" is false, sending msgs to a non-exist topic
> will trigger too many meta requests, everytime a metadata response is
> returned, because it does not contain the metadata for the topic, it's going
> to try again until TimeoutException is thrown;
> This is a waste and sometimes causes too much overhead when unexpected msgs
> are arrived.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
