Hi, Thank you for your answer @Matthias. Indeed, I need a kind of symmetric join. However, KStream-KStream join doesn't match with my use case: I will need to generate the events in the changelog (e.g a campaign marketing with a certain Id/Key) because they live only for the join in a defined windows. Let's say I got a click in the record stream and then the campaign entity arrive later on within the windows, the join enriched stream works. But after the windows, new click arrive in the record stream and won't be able to find the campaign entity since the windows containing the information has passed. I haven't tried KTable-KTable yet but I think my clicks for example doesn't really match a changelog stream in my opinion.
@Guozhang Ok so If I fake the timestamp in the record stream (KStream) (let's say add 1 day), I could manage to give one day chance to the late arrival in the changelog stream (my KTable) for the Join to be processed. Let me try. Thanks Regards, On Tue, Jul 19, 2016 at 11:45 PM, Guozhang Wang <wangg...@gmail.com> wrote: > Hello Nicolas, > > If this missing matched record issue is mainly due to the order these two > streams were processed (e.g., say your corresponding changelog record was a > bit late compared with the record stream's record with the same key), you > can try to "hint" Kafka Streams library to give the changelog stream a bit > more time ahead by specifying its timestamps using the TimestampExtractor > with an earlier value against the record stream. And Kafka Streams will do > a best-effort "stream synchronization" to make sure these two streams were > processed at roughly the same pace based on record timestamps, which will > result in records from the changelog stream to be processed in-priori to > the record stream. > > > Guozhang > > > On Tue, Jul 19, 2016 at 6:10 AM, Matthias J. Sax <matth...@confluent.io> > wrote: > > > Hi Nicolas, > > > > your are right, it is currently not possible to get a result from a > > KTable update (and this is by design). The idea is, that the KStream is > > enriched with the *current state* of KTable -- thus, for each KStream > > record a look-up in KTable is done. (In this sense, a KStream-KTable > > join in asymmetric.) > > > > If you need a symmetric join (ie, lookup for both directions), you can > > either use a KTable-KTable or KStream-KStream join. Not sure, if this > > might work for your use case. > > > > -Matthias > > > > > > On 07/19/2016 01:36 PM, Nicolas PHUNG wrote: > > > Hi, > > > > > > I'm using Kafka 0.10.0.0 with the Confluent platform 3.0.0 > > > > > > I manage to join a record stream (KStream / clicks stream) with a > > changelog > > > stream (KTable / an entity like a campaign related to a click for > > example). > > > When the entity in the KTable is inserted first (and the first time of > > > course) in Kafka, the record stream is processed as expected with the > > join > > > in a new enriched stream. This is good. > > > > > > My issue is when the record stream generate a record that contains a > > key/id > > > that hasn't been insert yet in the changelog stream/KTable. My process > > > generate a record stream without information in the enriched stream. > > Would > > > it be possible to recall this enriched stream process once the > changelog > > > record on my KTable received the missing id/key ? From my > understanding, > > > it's not possible right now to this with a KStream-KTable join. Is > there > > a > > > way to do something like this ? > > > > > > Thanks. > > > > > > Regards, > > > Nicolas PHUNG > > > > > > > > > > -- > -- Guozhang >
