If one of the broker we specify in the bootstrap servers list is down, there is a chance that the Producer (a brand new instance with no prior metadata) will never be able to publish anything to Kafka until that broker is up. Because the logic for getting the initial metadata is based on some random index to the set of bootstrap nodes and if it happens to be the down node, Kafka producer keeps on trying to get the metadata on that node only. It is never switched to another node. Without metadata, the Producer can never send anything.
The nodeIndexOffset is chosen at the creation of the NetworkClient (and this offset is not changed when we fail to get a new connection) and so for getting the metadata for the first time, there is a possibility that we keep on trying on the broker that is down. This can be a problem if a broker goes down and also a Producer is restarted or a new instance is brought up. Is this a known issue?
