Re: [idea] Kafka topic metadata

2021-06-15 Thread Ivan Yurchenko
Hi Garmes, I think it's worth writing a separate email for this as described in https://cwiki.apache.org/confluence/display/KAFKA/Kafka+Improvement+Proposals#KafkaImprovementProposals-WhoshouldinitiatetheKIP? (2nd paragraph). Best, Ivan On Tue, 15 Jun 2021 at 12:04, Garmes Amine wrote: > Hell

Re: [idea] Kafka topic metadata

2021-06-15 Thread Garmes Amine
Hello Israel, Ivan thank you for your feedback. Can I have the rights to create the KIP page please. Bests Garmes > On 14 Jun 2021, at 19:27, Israel Ekpo wrote: > > Sorry Ivan and Garmes, > > I misunderstood the suggestion earlier. I think this will be a great idea > for a KIP. > > https://

Re: [idea] Kafka topic metadata

2021-06-14 Thread Israel Ekpo
Sorry Ivan and Garmes, I misunderstood the suggestion earlier. I think this will be a great idea for a KIP. https://cwiki.apache.org/confluence/display/KAFKA/Kafka+Improvement+Proposals You were referring to metadata for the actual topic and not its contents. Sorry about that confusion. On M

Re: [idea] Kafka topic metadata

2021-06-14 Thread Ivan Yurchenko
Hi, Having metadata for topics seems pretty useful. Currently, one has to use external storage for this (e.g. a database) and the question of keeping topic and metadata in sync exists: A topic is deleted, how to delete its metadata? How to deal with delete-then-recreate scenarios (well, we have to

Re: [idea] Kafka topic metadata

2021-06-14 Thread Israel Ekpo
Garmes, I had similar questions in the past but @Matthias J. Sax pointed me to this https://cwiki.apache.org/confluence/display/KAFKA/KIP-244%3A+Add+Record+Header+support+to+Kafka+Streams+Processor+API With the headers, you can filter based on the header content and not just the contents of the

[idea] Kafka topic metadata

2021-06-11 Thread Garmes Amine
Hello Apache Kafka community, In the driection to make Kafka more cloud friendly, do you think it make sense to have key-velue metadata in the topic definition, this will allow to search, filter, select ... topics not only by name but also by metadata Ex: topic name: test topic metadata: