Re: Kafka Streams: Dynamic Topic Routing & Nonexistent Topics

2020-07-08 Thread Rhys Anthony McCaig
Guozhang - thank you for your thoughts. You are right - this is more about the producer client than the streams client. caching the metadata outside producer, e.g. in an admin client would not > be a perfect solution since in either way your metadata cache inside the > producer or inside the

Re: Kafka Streams: Dynamic Topic Routing & Nonexistent Topics

2020-07-04 Thread Guozhang Wang
Hello, Thanks for reaching out to the community for this. I think (maybe you've also suggested) it is rather an observation on producer client than on streams client. Generally speaking we want to know if we can fail fast if the metadata cannot be found in producer.send() call. And here are my

Kafka Streams: Dynamic Topic Routing & Nonexistent Topics

2020-07-01 Thread Rhys Anthony McCaig
Hi All, I have been recently working on a streams application that uses a TopicNameExtractor to dynamically route records based on the payload. This streams application is used by various other applications, and occasionally these other applications request for a record to be sent to a

Kafka Streams: Dynamic Topic Routing & Nonexistent Topics

2020-07-01 Thread Rhys Anthony McCaig
Hi All, I have been recently working on a streams application that uses a TopicNameExtractor to dynamically route records based on the payload. This streams application is used by various other applications, and occasionally these other applications request for a record to be sent to a