Re: Optimizing Kafka Streams Validation for Array Elements

Hi Bruno, Many thanks for trying to understand the problem, I prepare some repository which has simple demo https://github.com/pszymczyk/streams-flatmap. You can run /src/main/javaMain.main and then /src/test/java/SimpleTest, as the result you will have 6 messages on custom-repartition topic and 3

Re: Optimizing Kafka Streams Validation for Array Elements

I will provide you with the sample source code on GitHub today. Dnia 25 lutego 2025 20:47:36 CET, Bruno Cadonna napisał/a: >Hi Pawel, > >What is the "automatic json schema validation feature"? >Streams does not have such a thing built-in. > >Do you have an example of a third-party software that d

Re: Optimizing Kafka Streams Validation for Array Elements

Sorry for that, I should be more precise in that from the beginning. By automatic schema validation I mean SerDe combined with Default Deserialisation Exception Handler along with dlq topic as you can find in official streams documentation

Re: Optimizing Kafka Streams Validation for Array Elements

Hi Pawel, What is the "automatic json schema validation feature"? Streams does not have such a thing built-in. Do you have an example of a third-party software that does that validation? Regarding decomposing your input records into the smaller array elements, that should work with builder.s

Re: Optimizing Kafka Streams Validation for Array Elements

To make it clear I will give you an example: On the source topic we have: offset: 0 key: null value: { "request":{ }, "response":{ }, "data":[ { "deviceId":"23dc78ffa6c1ad3c038b", "event":{ "lvl":2, "someValue":{ "hori

Re: Optimizing Kafka Streams Validation for Array Elements

Ideal solution has two topics, can we somehow do flatMap and change a key without changing creating the internal repartition topic? This will allow us to skip using the internal repartition topic as source for k table. Dnia 25 lutego 2025 15:41:19 CET, Bruno Cadonna napisał/a: >Hi Pawel, > >I a

Re: Optimizing Kafka Streams Validation for Array Elements

Hi Pawel, I am not completely sure I understand the issue, because I am not a JSON expert. Is my understanding correct that the serde that you use to read from the input topic and from the repartition topic also do the validation of the JSON? Regarding point 2 and 3: I agree that the depen

Optimizing Kafka Streams Validation for Array Elements

Dear Kafka users, The last few days I spent working with Kafka Streams on some tasks which looked very easy at first glance but finally I struggled with the Streams Builder API and did something which I am not proud of. Please help me, I am open to any suggestions. On the input topic we have a mes