Hello kafka community, As I understand it, a kafka-streams join that involves a kTable: “the KTable lookup is done on the current KTable state, and thus, out-of-order records can yield non-deterministic result” [1]
Does the solution below involving an intermediate topic sound right to you? 1. kafka streams 1: - map topic1 in: key: the key as the join key between topic1 and topic2; value: topic1Or2UnionPayload, - map topic2 in key: the key as the join key between topic1 and topic2; value: topic1Or2UnionPayload, - merge mapped topics above into a single stream; the keys are identical for elements that need to be joined on both mapped topics/streams - write result to a new topic - 2. transformation part 2: (kafka consumer/processor or kafka streams if a suitable stateful transformation can be applied): - input the topic result from above with both mapped messages coming on a single pipe, linearised by join key - a state is need, likely a database (in case kafka-streams is applicable, great, it already embeds one) - when any two sides for the same join key are in the state, a pair can be emitted downstream - Does this make sense to you? Do you have any other experiences of possible approaches that you would like to share? Thank you [1] https://cwiki.apache.org/confluence/display/KAFKA/Kafka+Streams+Join+Semantics Dumitru-Nicolae Marasoui Software Engineer w kaluza.com <https://www.kaluza.com/> LinkedIn <https://www.linkedin.com/company/kaluza> | Twitter <https://twitter.com/Kaluza_tech> Kaluza Ltd. registered in England and Wales No. 08785057 VAT No. 100119879 Help save paper - do you need to print this email?
