Hi Matthias,
Originally I had written a really long answer against deprecations of the
method. When I finished writing, I realised that I actually agree with all
your points.
As you said, if a user is worried about how a table store is persisted, she
will almost always also be worried about
If you argue this way, would the consequence not be that we would need
to add even more overloads that allow to only use Materialized for the
subscription store?
-> I only want to switch to in-memory for the subscription store: Why do
I need to set Materialized for the main store?
Btw: the above
Hi Matthias,
Thanks for reviewing it. That's a nice question. I can provide you the
answer as a user of the API - but of course it very much depends on what
the original intention towards the user is.
The only reason I found out it was opinionated was when my tests using
in-memory materialization
I just re-read the KIP, and I am wondering why we would *only add* new
methods.
Is it an expected use case to only change the main stores but not the
subscription stores?
Wondering if we should deprecate the existing methods that only accept a
single `Materialized` parameter?
-Matthias
On
Hi everyone,
I am fine with sticking with Materialized and adding the info to the
javadoc then - so we keep the footprint smaller.
I will then update the KIP to match what we discussed here and send it for
a vote next week.
I will raise a new bug-fix ticket and change KAFKA-10383
Just catching up here.
I agree that we have two issue, and the first (align subscription store
to main store) can be done as a bug-fix.
For the KIP (that addressed the second), I tend to agree that reusing
`Materialized` might be better as it would keep the API surface area
smaller.
-Matthias
Hi Marco,
Just a quick clarification: I just reviewed the Materialized class. It looks
like the only undesirable members are:
1. Retention
2. Key/Value serdes
The underlying store type would be “KeyValueStore” , for which
case the withRetention javadoc already says it’s ignored.
Perhaps we
Hi John / Guozhang,
If I correctly understood John's message, he agrees on having the two
scenarios (piggy-back and api extension). In my view, these two scenarios
are separate tasks - the first one is a bug-fix and the second is an
improvement on the current API.
- bug-fix: On the current API,
Thanks Marco / John,
I think the arguments for not piggy-backing on the existing Materialized
makes sense; on the other hand, if we go this route should we just use a
separate Materialized than using an extended /
narrowed-scoped MaterializedSubscription since it seems we want to allow
users to
Thanks Marco,
Sorry if I caused any trouble!
I don’t remember what I was thinking before, but reasoning about it now, you
might need the fine-grained choice if:
1. The number or size of records in each partition of both tables is
small(ish), but the cardinality of the join is very high. Then
Hi Guozhang,
Apologies for the late answer. Originally that was my proposal - to
piggyback on the provided materialisation method (
https://issues.apache.org/jira/browse/KAFKA-10383).
John Roesler suggested to us to provide even further fine tuning on API
level parameters. Maybe we could see this
Thanks Marco,
Just a quick thought: what if we reuse the existing Materialized object for
both subscription and join stores, instead of introducing a new param /
class?
Guozhang
On Tue, Mar 2, 2021 at 1:07 AM Marco Aurélio Lotz
wrote:
> Hi folks,
>
> I would like to invite everyone to discuss
Hi folks,
I would like to invite everyone to discuss further KIP-718:
https://cwiki.apache.org/confluence/display/KAFKA/KIP-718%3A+Make+KTable+Join+on+Foreign+key+unopinionated
I welcome all feedback on it.
Kind Regards,
Marco Lotz
13 matches
Mail list logo
