Thanks Marco / John, I think the arguments for not piggy-backing on the existing Materialized makes sense; on the other hand, if we go this route should we just use a separate Materialized than using an extended / narrowed-scoped MaterializedSubscription since it seems we want to allow users to fully customize this store?
Guozhang On Thu, Apr 1, 2021 at 5:28 PM John Roesler <vvcep...@apache.org> wrote: > Thanks Marco, > > Sorry if I caused any trouble! > > I don’t remember what I was thinking before, but reasoning about it now, > you might need the fine-grained choice if: > > 1. The number or size of records in each partition of both tables is > small(ish), but the cardinality of the join is very high. Then you might > want an in-memory table store, but an on-disk subscription store. > > 2. The number or size of records is very large, but the join cardinality > is low. Then you might need an on-disk table store, but an in-memory > subscription store. > > 3. You might want a different kind (or differently configured) store for > the subscription store, since it’s access pattern is so different. > > If you buy these, it might be good to put the justification into the KIP. > > I’m in favor of the default you’ve proposed. > > Thanks, > John > > On Mon, Mar 29, 2021, at 04:24, Marco Aurélio Lotz wrote: > > Hi Guozhang, > > > > Apologies for the late answer. Originally that was my proposal - to > > piggyback on the provided materialisation method ( > > https://issues.apache.org/jira/browse/KAFKA-10383). > > John Roesler suggested to us to provide even further fine tuning on API > > level parameters. Maybe we could see this as two sides of the same coin: > > > > - On the current API, we change it to piggy back on the materialization > > method provided to the join store. > > - We extend the API to allow a user to fine tune different > materialization > > methods for subscription and join store. > > > > What do you think? > > > > Cheers, > > Marco > > > > On Thu, Mar 4, 2021 at 8:04 PM Guozhang Wang <wangg...@gmail.com> wrote: > > > > > Thanks Marco, > > > > > > Just a quick thought: what if we reuse the existing Materialized > object for > > > both subscription and join stores, instead of introducing a new param / > > > class? > > > > > > Guozhang > > > > > > On Tue, Mar 2, 2021 at 1:07 AM Marco Aurélio Lotz < > cont...@marcolotz.com> > > > wrote: > > > > > > > Hi folks, > > > > > > > > I would like to invite everyone to discuss further KIP-718: > > > > > > > > > > > > > > > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-718%3A+Make+KTable+Join+on+Foreign+key+unopinionated > > > > > > > > I welcome all feedback on it. > > > > > > > > Kind Regards, > > > > Marco Lotz > > > > > > > > > > > > > -- > > > -- Guozhang > > > > > > -- -- Guozhang
