Hi Luke and Kafka Dev Team, Any interest in making Kafka Producer non-blocking when Broker is down and when the metadata refresh cache does not have topic details? This seems to be a missing piece when it comes to Kafka Producer not being able to handle state when it is really down vs metadata refresh is not available.
I hope there is enough interest to make this producer broker down vs metadata not available. Thanks, Bhavesh On Mon, Oct 10, 2022 at 4:04 PM Bhavesh Mistry <mistry.p.bhav...@gmail.com> wrote: > Hi Luke, > > Thanks for the pointers. > > Sorry for being late I was out. > > > > I would like to propose the following which might be a little different > from the Old one: > > Kafka Producer must distinguish between *broker down state* vs *metadata > NOT available* for a given topic. > > > > Like the boot-strap server option, many applications (like ours) do not > dynamically create topics and publish/subscribe to predefine topics. So, > the Kafka producer can have a configuration option to “*predefine-topics*”. > When a predefine-topic is configured, Metadata is fetched for those > pre-defined topics before the producer is initialized. Also, these > pre-defined topics will always guarantee that Metadata will be refreshed > before it expires meaning (the metadata cache will expire at X time, then > the producer should automatically fetch metadata refresh request X-(3000) > ms so the cache will always have the latest mapping of topic partition > states continue to fetch everyone seconds till it expires cache X-2000 and > X-1000). This will guarantee the non-blocking behavior for pre-defined > topics. Blocking behavior is acceptable for topics that are NOT defined > ahead of time or dynamic topics. > > > > Another configuration we should have is to *drop-message-on-broker-down* > (true or false), even if the metadata has expired just DROP the message > till the broker is online. Do NOT block the application thread which puts > stuff on the Kafka in-memory queue. Of course, the Kafka producer will > have to keep track of all brokers and it states the in-memory data > structure and update it periodically (when send is a success or ping to > port (IP: port) is a success). > > > > Luke and others let me know what you think about it. > > > I can write documents if there is interest in the topic. > > > Thanks, > > > Bhavesh > > > On Sun, Sep 25, 2022 at 8:44 PM Luke Chen <show...@gmail.com> wrote: > >> Hi Bhavesh, >> >> I understand your point. >> There was an old KIP with the similar idea which was not accepted by the >> community in the end. >> Maybe you can try to bring it back to the community again, or try to >> propose your own KIP for this idea? >> >> https://cwiki.apache.org/confluence/display/KAFKA/KIP-286%3A+producer.send%28%29+should+not+block+on+metadata+update >> >> Thank you. >> Luke >> >> On Sat, Sep 24, 2022 at 6:36 AM Bhavesh Mistry < >> mistry.p.bhav...@gmail.com> >> wrote: >> >> > Hello Kafka Team, >> > >> > I would appreciate any insight into how to distinguish between Brocker >> Down >> > vs Metadata Refresh not available due to timing issues. >> > >> > Thanks, >> > >> > Bhavesh >> > >> > On Mon, Sep 19, 2022 at 12:50 PM Bhavesh Mistry < >> > mistry.p.bhav...@gmail.com> >> > wrote: >> > >> > > Hello Kafka Team, >> > > >> > > >> > > >> > > We have an environment where Kafka Broker can go down for whatever >> > reason. >> > > >> > > >> > > >> > > Hence, we had configured MAX_BLOCK_MS_CONFIG=0 because we wanted to >> drop >> > > messages when brokers were NOT available. >> > > >> > > >> > > >> > > Now the issue is we get data loss due to METADATA not being available >> and >> > > get this exception “*Topic <topic> not present in metadata after 0 >> ms.”. >> > > *This is due to the fast metadata has expired and the next request to >> > > send an event does not have metadata. >> > > >> > > >> > > >> > > Why does Kafka have his design? Why can’t Kafka distinguish between >> > > Broker down vs metadata refresh not available? Is it reasonable to >> > expect >> > > metadata would refresh BEFORE it expires so metadata refresh doesn’t >> need >> > > before it expires? Have Metadata ready before expires? Any particular >> > > reason send() has wait for metadata refresh vs background thread that >> > > automatically refreshes metadata before it expires, hence send() >> method >> > > never incur wait(). >> > > >> > > >> > > Let me know what suggestion you have to prevent the application thread >> > > from blocking (MAX_BLOCK_MS_CONFIG) when the Kafka brokers are DOWN vs >> > > metadata is NOT available due to expiration. >> > > >> > > >> > > >> > > Let me know your suggestions and what you think about metadata >> refresh. >> > > Should Kafka Producer be proactively refreshing metadata intelligently >> > > rather than what the producer does today? >> > > >> > > >> > > >> > > >> > > >> > > Thanks, >> > > Bhavesh >> > > >> > >> >
