Neha Narkhede created KAFKA-788:
-----------------------------------
Summary: Periodic refresh of topic metadata on the producer
doesn't include all topics
Key: KAFKA-788
URL: https://issues.apache.org/jira/browse/KAFKA-788
Project: Kafka
Issue Type: Bug
Components: producer
Affects Versions: 0.8
Reporter: Neha Narkhede
Assignee: Jun Rao
We added a patch to the producer to refresh the metadata for all topics
periodically. However, the producer only does this for the topics in the last
batch. But some topics sent by the producer could be low throughput and might
not be present in every batch. If we bounce the cluster or if brokers fail and
leaders change, the metadata for those low throughput topic is not refreshed by
this periodic topic metadata request. The next produce request for those topics
have to fail and then a separate metadata request needs to be reissued to
handle the produce request. This is especially a problem for the migration
tool. So even if the producer had a chance to refresh the metadata when the
leader changed, it throws LeaderNotAvailableExceptions much later when it sends
a request for that topic.
I propose we just fetch data for all topics sent by the producer in the
periodic refresh of topic metadata
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
