I've read that JIRA (although I don't understand every single thing), and I got the feeling it is not exactly the same problem. I am aware of the Global Tables, and I've tried that first, but I seem unable to do what I need to do.
I'm replicating a relational database, and on a one-to-many relationship I'd like to publish a joined message if either of the source streams receives an update. In the Global Table Wiki: https://cwiki.apache.org/confluence/display/KAFKA/KIP-99%3A+Add+Global+Tables+to+Kafka+Streams I see this: "The GlobalKTable will only be used for doing lookups. That is, data arriving in the GlobalKTable will not trigger the join. " So how would I go about doing this? regards, Frank On Tue, Feb 21, 2017 at 10:38 AM, Eno Thereska <eno.there...@gmail.com> wrote: > Hi Frank, > > As far as I know the design in that wiki has been superceded by the Global > KTables design which is now coming in 0.10.2. Hence, the JIRAs that are > mentioned there (like KAFKA-3705). There are some extensive comments in > https://issues.apache.org/jira/browse/KAFKA-3705 < > https://issues.apache.org/jira/browse/KAFKA-3705> illustrating why this > design is particularly challenging and why Global KTables was chosen > instead. I'm not sure if you still want to pursue that original design, > since it is not proven to work. > > Guozhang, perhaps we need to add a note saying that Global KTables is the > new design? > > Thanks > Eno > > > On 21 Feb 2017, at 07:35, Frank Lyaruu <flya...@gmail.com> wrote: > > > > Hi all, > > > > I'm trying to implement joining two Kafka tables using a 'remote' key, > > basically as described here: > > > > https://cwiki.apache.org/confluence/display/KAFKA/ > Discussion%3A+Non-key+KTable-KTable+Joins > > > > Under the "Implementation Details" there is one line I don't know how to > > do: > > > > > > 1. First of all, we will repartition this KTable's stream, by key > > computed from the *mapper(K, V) → K1*, so that it is co-partitioned by > > the same key. The co-partition topic is partitioned on the new key, > but the > > message key and value are unchanged, and log compaction is turned off. > > > > > > How do I do that? I've been unable to find any documentation, I've looked > > at the StreamPartitionAssignor, that seems relevant, but I could use some > > help. Does anyone have an example? > > > > regards, Frank > >
