Michael,
We can only support outerJoin if both tables are keyed the same way. Lets
say for example you can map both ways, however, the key for each table is
of a different type. So t1 is long and t2 is string - what is the key type
of the resulting GlobalKTable? So when you subsequently join to this table,
and do a lookup on it, which key are you using?
Thanks,
Damian
On Wed, 7 Dec 2016 at 14:31 Michael Noll <mich...@confluent.io> wrote:
> Damian,
>
> yes, that makes sense.
>
> But I am still wondering: In your example, there's no prior knowledge "can
> I map from t1->t2" that Streams can leverage for joining t1 and t2 other
> than blindly relying on the user to provide an appropriate KeyValueMapper
> for K1/V1 of t1 -> K2/V2 of t2. In other words, if we allow the user to
> provide a KeyValueMapper from t1->t2 (Streams does not know at compile time
> whether this mapping will actually work), then we can also allow the user
> to provide a corresponding "reverse" mapper from t2->t1. That is, we could
> say that an outer join between two global tables IS supported, but if and
> only if the user provides two KeyValueMappers, one for t1->t2 and one for
> t2->t1.
>
> The left join t1->t2 (which is supported in the KIP), in general, works
> only because of the existence of the user-provided KeyValueMapper from
> t1->t2. The outer join, as you point out, cannot satisfied as easily
> because Streams must know two mappers, t1->t2 plus t2->t1 -- otherwise the
> outer join won't work.
>
>
>
>
>
> On Wed, Dec 7, 2016 at 3:04 PM, Damian Guy <damian....@gmail.com> wrote:
>
> > Hi Michael,
> >
> > Sure. Say we have 2 input topics t1 & t2 below:
> > t1{
> > int key;
> > string t2_id;
> > ...
> > }
> >
> > t2 {
> > string key;
> > ..
> > }
> > If we create global tables out of these we'd get:
> > GlobalKTable<Integer, ?> t1;
> > GlobalKTable<String, ?> t2;
> >
> > So the join can only go in 1 direction, i.e, from t1 -> t2 as in order to
> > perform the join we need to use a KeyValueMapper to extract the t2 key
> from
> > the t1 value.
> >
> > Does that make sense?
> >
> > Thanks,
> > Damian
> >
> > On Wed, 7 Dec 2016 at 10:44 Michael Noll <mich...@confluent.io> wrote:
> >
> > > > There is no outer-join for GlobalKTables as the tables may be keyed
> > > > differently. So you need to use the key from the left side of the
> join
> > > > along with the KeyValueMapper to resolve the right side of the join.
> > This
> > > > wont work the other way around.
> > >
> > > Care to elaborate why it won't work the other way around? If, for
> > example,
> > > we swapped the call from leftTable.join(rightTable) to
> > > rightTable.join(leftTable), that join would work, too. Perhaps I am
> > > missing something though. :-)
> > >
> > >
> > >
> > >
> > > On Wed, Dec 7, 2016 at 10:39 AM, Damian Guy <damian....@gmail.com>
> > wrote:
> > >
> > > > Hi Matthias,
> > > >
> > > > Thanks for the feedback.
> > > >
> > > > There is no outer-join for GlobalKTables as the tables may be keyed
> > > > differently. So you need to use the key from the left side of the
> join
> > > > along with the KeyValueMapper to resolve the right side of the join.
> > This
> > > > wont work the other way around.
> > > >
> > > > On the bootstrapping concern. If the application is failing before
> > > > bootstrapping finishes, the problem is likely to be related to a
> > terminal
> > > > exception, i.e., running out of disk space, corrupt state stores etc.
> > In
> > > > these cases, we wouldn't want the application to continue. So i think
> > > this
> > > > is ok.
> > > >
> > > > Thanks,
> > > > Damian
> > > >
> > > > On Tue, 6 Dec 2016 at 21:56 Matthias J. Sax <matth...@confluent.io>
> > > wrote:
> > > >
> > > > > Thanks for the KIP Damian. Very nice motivating example!
> > > > >
> > > > > A few comments:
> > > > >
> > > > > - why is there no outer-join for GlobalKTables
> > > > > - on bootstrapping GlobalKTable, could it happen that this never
> > > > > finishes if the application fails before bootstrapping finishes and
> > new
> > > > > data gets written at the same time? Do we need to guard against
> this
> > > > > (seems to be a very rare corner case, so maybe not required)?
> > > > >
> > > > >
> > > > > -Matthias
> > > > >
> > > > >
> > > > > On 12/6/16 2:09 AM, Damian Guy wrote:
> > > > > > Hi all,
> > > > > >
> > > > > > I would like to start the discussion on KIP-99:
> > > > > >
> > > > > https://cwiki.apache.org/confluence/pages/viewpage.
> > > > action?pageId=67633649
> > > > > >
> > > > > > Looking forward to your feedback.
> > > > > >
> > > > > > Thanks,
> > > > > > Damian
> > > > > >
> > > > >
> > > > >
> > > >
> > >
> >
>
>
>
> --
> *Michael G. Noll*
> Product Manager | Confluent
> +1 650 453 5860 <(650)%20453-5860> | @miguno <https://twitter.com/miguno>
> Follow us: Twitter <https://twitter.com/ConfluentInc> | Blog
> <http://www.confluent.io/blog>
>
