Re: [VOTE] KIP-150 - Kafka-Streams Cogroup

wrote: > > > Thanks for the KIP. > > +1 (binding) > > > > -Bill > > > > > > On Wed, Nov 6, 2019 at 1:22 AM Matthias J. Sax > > wrote: > > > > > +1 (binding) > > > > > > > > > On 10/31/19 10:52 AM, Walker Carls

Re: [VOTE] KIP-150 - Kafka-Streams Cogroup

/19 10:52 AM, Walker Carlson wrote: > > > Hello all, > > > > > > I'd like to call a vote on the updated KIP-150: Kafka-Streams Cogroup > > > found here > > > < > > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-150+-+Kafka-Streams+Cogroup > > > > > > > > > Thanks, > > > Walker > > > > > > > > -- -- Guozhang

Re: [VOTE] KIP-150 - Kafka-Streams Cogroup

Thanks for the KIP. +1 (binding) -Bill On Wed, Nov 6, 2019 at 1:22 AM Matthias J. Sax wrote: > +1 (binding) > > > On 10/31/19 10:52 AM, Walker Carlson wrote: > > Hello all, > > > > I'd like to call a vote on the updated KIP-150: Kafka-Streams Cogrou

Re: [VOTE] KIP-150 - Kafka-Streams Cogroup

+1 (binding) On 10/31/19 10:52 AM, Walker Carlson wrote: > Hello all, > > I'd like to call a vote on the updated KIP-150: Kafka-Streams Cogroup > found here > <https://cwiki.apache.org/confluence/display/KAFKA/KIP-150+-+Kafka-Streams+Cogroup> > > Thanks, &

Re: [DISCUSS] KIP-150 - Kafka-Streams Cogroup

>>>>>>>> > > >>>>>>>>>> On Wed, Oct 23, 2019 at 9:58 PM Sophie Blee-Goldman < > > >>>>>>>> sop...@confluent.io > > >>>>>>>>>> > > >>>>>>>>>> wrote: > > >>>>>>>>>>

[VOTE] KIP-150 - Kafka-Streams Cogroup

Hello all, I'd like to call a vote on the updated KIP-150: Kafka-Streams Cogroup found here <https://cwiki.apache.org/confluence/display/KAFKA/KIP-150+-+Kafka-Streams+Cogroup> Thanks, Walker

Re: [DISCUSS] KIP-150 - Kafka-Streams Cogroup

ion I have then is about the > >>>>>>> operator/store/repartition > >>>>>>>>>>> naming -- seems like > >>>>>>>>>>> we can name the underlying store/changelog through the > >>>>>>

Re: [DISCUSS] KIP-150 - Kafka-Streams Cogroup

>> >>>>>>>>>>>> Interesting idea, Sophie. >>>>>>>>>>>> >>>>>>>>>>>> So far, we tried to reuse existing config objects and >> only >>>> add >>>>>

Re: [DISCUSS] KIP-150 - Kafka-Streams Cogroup

t;> maybe the > > > > > > > > >>>> answer seems obviously "no") but we seem to often end up > > > needing > > > > > > to > > > > > > > > add > > > > > > > > >>> new > > > > > > > > >>>> overloads and/or deprecate old ones as new features or > > > > > > requirements > > > > > > > > >> come > > &g

Re: [DISCUSS] KIP-150 - Kafka-Streams Cogroup

e an initializer and a materialized. > > If we > > > > > > ever > > > > > > > >>> need > > > > > > > >>>> to add > > > > > > > >>>> a new para

Re: [DISCUSS] KIP-150 - Kafka-Streams Cogroup

t; > > > > > >>> > > > > > > >>>> wrote: > > > > > > >>>> > > > > > > >>>>> Hi Sophie, > > > > > > >>>>> > > > > > > >>>>> Thank

Re: [DISCUSS] KIP-150 - Kafka-Streams Cogroup

> > > However > > > > > >> each > > > > > >>>>> subsequent stream-aggregator pair is added on to a cogroup > > stream > > > > so > > > > > >> it > > > > > >>>>> needs both arg

Re: [DISCUSS] KIP-150 - Kafka-Streams Cogroup

rlapping key spaces > and > > > any > > > > >>> kind > > > > >>>>> of value types. Once aggregated its value will be reduced into > > one > > > > >> type. > > > > >>>>> This is why you need only on

Re: [DISCUSS] KIP-150 - Kafka-Streams Cogroup

t;>>> can collect many grouped streams with overlapping key spaces > and > > > any > > > > >>> kind > > > > >>>>> of value types. Once aggregated its value will be reduced into > > one > > > > >> type. > > > > >>>>> This is why you need only o

Re: [DISCUSS] KIP-150 - Kafka-Streams Cogroup

, > > > >>>>> Walker > > > >>>>> > > > >>>>> On Tue, Oct 22, 2019 at 8:59 PM Sophie Blee-Goldman < > > > >>> sop...@confluent.io> > > > >>>>> wrote:

Re: [DISCUSS] KIP-150 - Kafka-Streams Cogroup

; >>>>> This is a good question and I will include this explanation in > the > > > kip > > > >>> as > > > >>>>> well. > > > >>>>> > > > >>>>> Thanks, > > > >>>>> Walker > > > >>

Re: [DISCUSS] KIP-150 - Kafka-Streams Cogroup

awkward to me that with the current API, we > > >> have a > > >>>>>> nearly identical > > >>>>>> "add stream to cogroup" method, except for the first which has a > > >>>>> different > > >>>>>> signature > > >>&g

Re: [DISCUSS] KIP-150 - Kafka-Streams Cogroup

tor) ). I'm not sure what it > >>> would > >>>>>> look like exactly, > >>>>>> but I was just wondering if you'd considered and/or rejected any > >>>>>> alternative APIs? > >>>>>> > >>>>>> 2) Th

Re: [DISCUSS] KIP-150 - Kafka-Streams Cogroup

t; look like exactly, > >>>>>> but I was just wondering if you'd considered and/or rejected any > >>>>>> alternative APIs? > >>>>>> > >>>>>> 2) This might just be my lack of familiarity with "cogroup" as a > >>> concep

Re: [DISCUSS] KIP-150 - Kafka-Streams Cogroup

gt; how >>>>>> exactly >>>>>> the different streams are joined through the ValueJoiners. Would this >>> new >>>>>> cogroup >>>>>> simply concatenate the values from the different cogroup streams, or >>>>> c

Re: [DISCUSS] KIP-150 - Kafka-Streams Cogroup

;>> aren't fluent > > >>> in cogroup semantics :) > > >>> > > >>> Cheers, > > >>> Sophie > > >>> > > >>> On Thu, Oct 17, 2019 at 3:06 PM Walker Carlson < > wcarl...@confluent.io> > > >>>

Re: [DISCUSS] KIP-150 - Kafka-Streams Cogroup

>>> in cogroup semantics :) > >>> > >>> Cheers, > >>> Sophie > >>> > >>> On Thu, Oct 17, 2019 at 3:06 PM Walker Carlson > >>> wrote: > >>> > >>>> Good catch I updated that. > >>>>

Re: [DISCUSS] KIP-150 - Kafka-Streams Cogroup

e made a PR for this KIP >>>> >>>> I then am splitting it into 3 parts, first cogroup for a key-value >> store >>> ( >>>> here <https://github.com/apache/kafka/pull/7538>), then for a >>>> timeWindowedStore, and then a sessionWin

Re: [DISCUSS] KIP-150 - Kafka-Streams Cogroup

; > wrote: > > > > > > > Walker, > > > > > > > > thanks for picking up the KIP and reworking it for the changed API. > > > > > > > > Overall, the updated API suggestions make sense to me. The seem to > > ali

Re: [DISCUSS] KIP-150 - Kafka-Streams Cogroup

> > > > Walker > > > > > > On Tue, Oct 15, 2019 at 12:47 PM Matthias J. Sax > > > wrote: > > > > > > > Walker, > > > > > > > > thanks for picking up the KIP and reworking it for the changed API. > > > > >

Re: [DISCUSS] KIP-150 - Kafka-Streams Cogroup

> > > > > > Overall, the updated API suggestions make sense to me. The seem to > align > > > quite nicely with our current API design. > > > > > > One nit: In `CogroupedKStream#aggregate(...)` the type parameter of > > > `Materialized` sh

Re: [DISCUSS] KIP-150 - Kafka-Streams Cogroup

em to align > > quite nicely with our current API design. > > > > One nit: In `CogroupedKStream#aggregate(...)` the type parameter of > > `Materialized` should be `V`, not `VR`? > > > > > > -Matthias > > > > > > > > On

Re: [DISCUSS] KIP-150 - Kafka-Streams Cogroup

In `CogroupedKStream#aggregate(...)` the type parameter of > `Materialized` should be `V`, not `VR`? > > > -Matthias > > > > On 10/14/19 2:57 PM, Walker Carlson wrote: > > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-150+-+Kafka-Streams+Cogroup > > here

Re: [DISCUSS] KIP-150 - Kafka-Streams Cogroup

`, not `VR`? -Matthias On 10/14/19 2:57 PM, Walker Carlson wrote: > https://cwiki.apache.org/confluence/display/KAFKA/KIP-150+-+Kafka-Streams+Cogroup > here > is a link > > On Mon, Oct 14, 2019 at 2:52 PM Walker Carlson > wrote: > >> Hello all, >> >> I

Re: [DISCUSS] KIP-150 - Kafka-Streams Cogroup

https://cwiki.apache.org/confluence/display/KAFKA/KIP-150+-+Kafka-Streams+Cogroup here is a link On Mon, Oct 14, 2019 at 2:52 PM Walker Carlson wrote: > Hello all, > > I have picked up and updated KIP-150. Due to changes to the project since > KIP #150 was written there are

[DISCUSS] KIP-150 - Kafka-Streams Cogroup

Hello all, I have picked up and updated KIP-150. Due to changes to the project since KIP #150 was written there are a few items that need to be updated. First item that changed is the adoption of the Materialized parameter. The second item is the WindowedBy. How the old KIP handles windowing is

Re: [Vote] KIP-150 - Kafka-Streams Cogroup

; > > > > > > >> > You will firs see > > > > > > > >> > > > > > > > > >> > 1L, Customer[ > > > > > > > >> > > > > > > > > >> > cart:{Ite

Re: [Vote] KIP-150 - Kafka-Streams Cogroup

> > > > > > > >> > cart:{Item[no:01]}, > > > > > > >> > purchases:{}, > > > > > > >> > wishList:{} > > > > > > >> > ] > > > > > > >&

Re: [DISCUSS] KIP-150 - Kafka-Streams Cogroup

The KIP and PR have been updated please go take a look and vote. For those worried about the [DISCUSS] Streams DSL/StateStore Refactoring email thread affecting this I believe the cogroup methods fit well into the streams dsl and won't need to change. We can update the aggregate methods in the

Re: [DISCUSS] KIP-150 - Kafka-Streams Cogroup

+1 Thanks, Bill On Wed, Jun 14, 2017 at 8:10 PM, Xavier Léauté wrote: > +1 from me > > any stream should be able to initialize the cogroup > > On Wed, Jun 14, 2017 at 3:44 PM Kyle Winkelman > wrote: > > > I will update the kip to have only the

Re: [DISCUSS] KIP-150 - Kafka-Streams Cogroup

+1 from me any stream should be able to initialize the cogroup On Wed, Jun 14, 2017 at 3:44 PM Kyle Winkelman wrote: > I will update the kip to have only the aggregator in the first cogroup call > and the initializer and serde in the aggregate calls. > > This seems to

Re: [DISCUSS] KIP-150 - Kafka-Streams Cogroup

I will update the kip to have only the aggregator in the first cogroup call and the initializer and serde in the aggregate calls. This seems to be the consensus on the email chain. Thanks, Kyle On Jun 14, 2017 5:41 PM, wrote: That is not the case. No matter which stream the record comes in on

Re: [DISCUSS] KIP-150 - Kafka-Streams Cogroup

That is not the case. No matter which stream the record comes in on the initializer is called if there is not currently an object in the store. On Jun 14, 2017 5:12 PM, "Guozhang Wang" wrote: > While regarding where we should ask users to set serdes: I think I'm > convinced

Re: [DISCUSS] KIP-150 - Kafka-Streams Cogroup

While regarding where we should ask users to set serdes: I think I'm convinced by Xavier that they should be in the `aggregate` call (but only those does not pass in a state store supplier) instead of the `KStream#cogroup` call to be consistent with other aggregate functions. BTW another

Re: [DISCUSS] KIP-150 - Kafka-Streams Cogroup

To clarify it isn't required to have the initializer in the first cogroup because the first aggregator will have the value type. I like how the initializer makes it abundantly clear that the final type will be that. Right now I'm split because the case may be made that you want to supply a

Re: [DISCUSS] KIP-150 - Kafka-Streams Cogroup

I'd suggest we do not block this KIP until the serde work has been sorted out: we cannot estimate yet how long it will take yet. Instead let's say make an agreement on where we want to specify the serdes: whether on the first co-group call or on the aggregate call. Also about the initializer

Re: [DISCUSS] KIP-150 - Kafka-Streams Cogroup

+1 on deferring discussion on Serdes until API improvements are ironed out. On Tue, Jun 13, 2017 at 2:06 PM, Matthias J. Sax wrote: > Hi, > > I am just catching up on this thread. (1) as most people agree, we > should not add anything to KStreamBuilder (btw: we actually

Re: [DISCUSS] KIP-150 - Kafka-Streams Cogroup

Hi, I am just catching up on this thread. (1) as most people agree, we should not add anything to KStreamBuilder (btw: we actually plan to move #merge() to KStream and deprecate it on KStreamBuilder as it's a quite unnatural API atm). About specifying Serdes: there is still the idea to improve

Re: [DISCUSS] KIP-150 - Kafka-Streams Cogroup

You're right, I haven't thought of that. Cheers, Michał On 13/06/17 13:00, Kyle Winkelman wrote: First, I would prefer not calling it aggregate because there are already plenty of aggregate methods. Second, I dont think this would really work because after each aggregate you now have a

Re: [DISCUSS] KIP-150 - Kafka-Streams Cogroup

First, I would prefer not calling it aggregate because there are already plenty of aggregate methods. Second, I dont think this would really work because after each aggregate you now have a unique KTable (someone may want a table with 4 streams and reuse those 4 in another table but with one more

Re: [DISCUSS] KIP-150 - Kafka-Streams Cogroup

I think we are discussing two separate things here, so it might be worth clarifying: 1) the position of the initializer with respect to the aggregators. If I understand correctly, Guozhang seems to think it is more natural to specify the initializer first, despite it not bearing any relation to

Re: [Vote] KIP-150 - Kafka-Streams Cogroup

:{Item[no:01]}, > > > > > >> > purchases:{Item[no:07],Item[no:08]}, > > > > > >> > > > > > > >> > wishList:{} > > &g

Re: [Vote] KIP-150 - Kafka-Streams Cogroup

purchases:{Item[no:07],Item[no:08]}, > > > > >> > > > > > >> > wishList:{} > > > > >> > ] > > > > >> > > > > > >> > 1L, Customer[ &

Re: [DISCUSS] KIP-150 - Kafka-Streams Cogroup

>>>>>>> > > >> > > >>>>>>>>>>>>> - minor: could you add an exact example (similar to > > what > > >> > > >>>>> Jay’s > > >> > > >&g

Re: [DISCUSS] KIP-150 - Kafka-Streams Cogroup

>>>>>> - minor: could you add an exact example (similar to > > what > > >> > > >>>>> Jay’s > > >> > > >>>>>>>>> example > > >> > > >>>>>>>>>&

Re: [DISCUSS] KIP-150 - Kafka-Streams Cogroup

an optimizer would take the existing > DSL > >> > > >>>>> and do > >> > > >>>>>>>> the > >> > > >>>>>>>>>>> right > >> > > >>>>&g

Re: [Vote] KIP-150 - Kafka-Streams Cogroup

> > > > >> > cart:{Item[no:01]}, > > > >> > purchases:{Item[no:07],Item[no:08]}, > > > >> > > > > >> > wishList:{} > > > >> >

Re: [DISCUSS] KIP-150 - Kafka-Streams Cogroup

> > >>>>> like >> > > >>>>>> to >> > > >>>>>>>>> hear >> > > >>>>>>>>>>> your >> > > >>>>>>>>>>>>> thoughts on whether it’s possib

Re: [DISCUSS] KIP-150 - Kafka-Streams Cogroup

t;>>>>>> On May 5, 2017, at 4:39 PM, Jay Kreps < > > > >> j...@confluent.io> > > > >>>>>> wrote: > > > >>>>>>>>>>>>>> > > > >>>>>>&

Re: [DISCUSS] KIP-150 - Kafka-Streams Cogroup

> > >>>>>>>>>>>>>> we often use. In that use case you have a dozen systems > > >>>>> each > > >>>>>> of > > >>>>>>>>> which > > >>>>>>>>>>>

Re: [Vote] KIP-150 - Kafka-Streams Cogroup

]}, > > >> > > > >> > wishList:{} > > >> > ] > > >> > > > >> > ... > > >> > > > >> > > > >> > I'm wondering if it makes more sense to only start sending the > update > > if > > >> > the corresponding agg-key has seen at least one input from each of > the > > >> > input stream? Maybe it is out of the scope of this KIP and we can > make > > >> it a > > >> > more general discussion in a separate one. > > >> > > > >> > > > >> > Guozhang > > >> > > > >> > > > >> > On Fri, May 19, 2017 at 8:37 AM, Xavier Léauté <xav...@confluent.io > > > > >> > wrote: > > >> > > > >> > > Hi Kyle, I left a few more comments in the discussion thread, if > you > > >> > > wouldn't mind taking a look > > >> > > > > >> > > On Fri, May 19, 2017 at 5:31 AM Kyle Winkelman < > > >> winkelman.k...@gmail.com > > >> > > > > >> > > wrote: > > >> > > > > >> > > > Hello all, > > >> > > > > > >> > > > I would like to start the vote on KIP-150. > > >> > > > > > >> > > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-150+-+ > > >> > > Kafka-Streams+Cogroup > > >> > > > > > >> > > > Thanks, > > >> > > > Kyle > > >> > > > > > >> > > > > >> > > > >> > > > >> > > > >> > -- > > >> > -- Guozhang > > >> > > > >> > > > > > >

Re: [DISCUSS] KIP-150 - Kafka-Streams Cogroup

for each > >>>>>> customer > >>>>>>>>> that > >>>>>>>>>>> has > >>>>>>>>>>>>>> all the relevant info in a usable form and is > >> up-to-date > >

Re: [DISCUSS] KIP-150 - Kafka-Streams Cogroup

>>>>>>>>>> already >>>>>>>>>>>>>> co-partitioned. A distributed database would do this >>> under >>>>> the >>>>>>>>> covers >>>>>>>>>>>>> which >>>>>>>>

Re: [DISCUSS] KIP-150 - Kafka-Streams Cogroup

t;> but it > > >> > >> > seems > > >> > >> > > > like > > >> > >> > > > >>>> it > > >> > >> > > > >>>>>

Re: [DISCUSS] KIP-150 - Kafka-Streams Cogroup

t; > >> > > > >>>>>> KTable<K, V3> table3 = >> > >> > > > >>>>>> builder.stream("topic3").groupByKey().aggregate( >> > initializer3 >> > >> , >> > >> > > > >>>>> ag

Re: [DISCUSS] KIP-150 - Kafka-Streams Cogroup

ate > > >> > > > states > > >> > > > >>>>> such > > >> > > > >>>>>> as Topic1Map, Topic2Map, and Topic3Map and the joiners > use > > >> those > > >> > &g

Re: [DISCUSS] KIP-150 - Kafka-Streams Cogroup

>>> produce this in the form of the first intermediate value > and > >> get > >> > > > sent > >> > > > >>>>>> through a KTableKTableOuterJoin where it will look up the > >> > current > >> > > > >> value

Re: [Vote] KIP-150 - Kafka-Streams Cogroup

{} > >> > ] > >> > > >> > ... > >> > > >> > > >> > I'm wondering if it makes more sense to only start sending the update > if > >> > the corresponding agg-key has seen at least one input from each of t

Re: [DISCUSS] KIP-150 - Kafka-Streams Cogroup

t; If you think through all possibilities for incoming topics >> you >> > > will >> > > > >> see >> > > > >>>>>> that no matter which topic it comes in through all three >> stores >

Re: [Vote] KIP-150 - Kafka-Streams Cogroup

> the corresponding agg-key has seen at least one input from each of the >> > input stream? Maybe it is out of the scope of this KIP and we can make >> it a >> > more general discussion in a separate one. >> > >> &g

Re: [Vote] KIP-150 - Kafka-Streams Cogroup

> wrote: > > > > > Hi Kyle, I left a few more comments in the discussion thread, if you > > > wouldn't mind taking a look > > > > > > On Fri, May 19, 2017 at 5:31 AM Kyle Winkelman < > winkelman.k...@gmail.com > > > > > > wrote: > > > > > > > Hello all, > > > > > > > > I would like to start the vote on KIP-150. > > > > > > > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-150+-+ > > > Kafka-Streams+Cogroup > > > > > > > > Thanks, > > > > Kyle > > > > > > > > > > > > > > > -- > > -- Guozhang > > >

Re: [Vote] KIP-150 - Kafka-Streams Cogroup

Fri, May 19, 2017 at 5:31 AM Kyle Winkelman <winkelman.k...@gmail.com > > > > wrote: > > > > > Hello all, > > > > > > I would like to start the vote on KIP-150. > > > > > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-150+-+ > > Kafka-Streams+Cogroup > > > > > > Thanks, > > > Kyle > > > > > > > > > -- > -- Guozhang >

Re: [DISCUSS] KIP-150 - Kafka-Streams Cogroup

me3 and use the second joiner to build the final >> > aggregate >> > > > >>>> value. >> > > > >>>>>> >> > > > >>>>>> If you think through all possibilities for incoming topics >> you >> > > will

Re: [Vote] KIP-150 - Kafka-Streams Cogroup

like to start the vote on KIP-150. > > > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-150+-+ > Kafka-Streams+Cogroup > > > > Thanks, > > Kyle > > > -- -- Guozhang

Re: [DISCUSS] KIP-150 - Kafka-Streams Cogroup

;K, V1> grouped1 = builder.stream("topic1"). > > > > >>>> groupByKey(); > > > > >>>>>> KGroupedStream<K, V2> grouped2 = builder.stream("topic2"). > > > > >>>> groupByKey(); > > > > >>

Re: [Vote] KIP-150 - Kafka-Streams Cogroup

nce/display/KAFKA/KIP-150+-+Kafka-Streams+Cogroup > > Thanks, > Kyle >

Re: [DISCUSS] KIP-150 - Kafka-Streams Cogroup

. > It > > >>>> will > > >>>>>> add its incoming object to the aggregate, update the store and > pass > > >> the > > >>>>> new > > >>>>>> aggregate on. This new aggregate goes through the KStreamCog

[Vote] KIP-150 - Kafka-Streams Cogroup

Hello all, I would like to start the vote on KIP-150. https://cwiki.apache.org/confluence/display/KAFKA/KIP-150+-+Kafka-Streams+Cogroup Thanks, Kyle

Re: [DISCUSS] KIP-150 - Kafka-Streams Cogroup

>>>>>>>>>>> the final aggregate CG value. This is something the user >> could >>>>>>> avoid >>>>>>>>>>>>> thinking about with this KIP. >>>>>>>>>>>>>

Re: [DISCUSS] KIP-150 - Kafka-Streams Cogroup

>> the > > > >>>>>>> key > > > >>>>>>>> in > > > >>>>>>>>> storeName3 and use the second joiner to build the final > > aggregate > > > >>>>>>&

Re: [DISCUSS] KIP-150 - Kafka-Streams Cogroup

ied > > >>>>>>>>> and all of the joiners must get used. > > >>>>>>>>> > > >>>>>>>>> Topology wise for N incoming streams this creates N > > >>>>>>>>> KStreamAggregates, 2*(N-1) KTab

Re: [DISCUSS] KIP-150 - Kafka-Streams Cogroup

>>>>>>>> queried > > >>>>>>>>> and all of the joiners must get used. > > >>>>>>>>> > > >>>>>>>>> Topology wise for N incoming streams this creates N > > >>>>>>>>> KStream

Re: [DISCUSS] KIP-150 - Kafka-Streams Cogroup

Stream<K, V2> grouped2 = builder.stream("topic2"). > >>>>>>> groupByKey(); > >>>>>>>>> KGroupedStream<K, V3> grouped3 = builder.stream("topic3"). > >>>>>>> groupByKey(); > >>>>>>>>> KTable<K, CG> cogrouped = grouped1.cogroup(initializer1, > >>> aggregator1, > >>>>>>>>> aggValueSerde1, storeName1) > >>>>>>>>> .cogroup(grouped2, aggregator2) > >>>>>>>>> .cogroup(grouped3, aggregator3) > >>>>>>>>> .aggregate(); > >>>>>>>>> > >>>>>>>>> As you can see this creates 1 StateStore, requires 1 initializer, > >>> and > >>>>> 1 > >>>>>>>>> aggValueSerde. The user no longer has to worry about the > >>> intermediate > >>>>>>>>> values and the joiners. All they have to think about is how each > >>>>> stream > >>>>>>>>> impacts the creation of the final CG object. > >>>>>>>>> > >>>>>>>>> When a new input arrives lets say at "topic1" it will first go > >>> through > >>>>>>> a > >>>>>>>>> KStreamAggreagte and grab the current aggregate from storeName1. > It > >>>>>>> will > >>>>>>>>> add its incoming object to the aggregate, update the store and > pass > >>>>> the > >>>>>>>> new > >>>>>>>>> aggregate on. This new aggregate goes through the KStreamCogroup > >>> which > >>>>>>> is > >>>>>>>>> pretty much just a pass through processor and you are done. > >>>>>>>>> > >>>>>>>>> Topology wise for N incoming streams the new api will only every > >>>>>>> create N > >>>>>>>>> KStreamAggregates and 1 KStreamCogroup. > >>>>>>>>> > >>>>>>>>> On Thu, May 4, 2017 at 4:42 PM, Matthias J. Sax < > >>>>> matth...@confluent.io > >>>>>>>> > >>>>>>>>> wrote: > >>>>>>>>> > >>>>>>>>>> Kyle, > >>>>>>>>>> > >>>>>>>>>> thanks a lot for the KIP. Maybe I am a little slow, but I could > >>> not > >>>>>>>>>> follow completely. Could you maybe add a more concrete example, > >>> like > >>>>>>> 3 > >>>>>>>>>> streams with 3 records each (plus expected result), and show the > >>>>>>>>>> difference between current way to to implement it and the > proposed > >>>>>>> API? > >>>>>>>>>> This could also cover the internal processing to see what store > >>> calls > >>>>>>>>>> would be required for both approaches etc. > >>>>>>>>>> > >>>>>>>>>> I think, it's pretty advanced stuff you propose, and it would > >>> help to > >>>>>>>>>> understand it better. > >>>>>>>>>> > >>>>>>>>>> Thanks a lot! > >>>>>>>>>> > >>>>>>>>>> > >>>>>>>>>> -Matthias > >>>>>>>>>> > >>>>>>>>>> > >>>>>>>>>> > >>>>>>>>>> On 5/4/17 11:39 AM, Kyle Winkelman wrote: > >>>>>>>>>>> I have made a pull request. It can be found here. > >>>>>>>>>>> > >>>>>>>>>>> https://github.com/apache/kafka/pull/2975 > >>>>>>>>>>> > >>>>>>>>>>> I plan to write some more unit tests for my classes and get > >>> around > >>>>>>> to > >>>>>>>>>>> writing documentation for the public api additions. > >>>>>>>>>>> > >>>>>>>>>>> One thing I was curious about is during the > >>>>>>>>>> KCogroupedStreamImpl#aggregate > >>>>>>>>>>> method I pass null to the KGroupedStream#repartitionIfRequired > >>>>>>>> method. > >>>>>>>>> I > >>>>>>>>>>> can't supply the store name because if more than one grouped > >>> stream > >>>>>>>>>>> repartitions an error is thrown. Is there some name that > someone > >>>>>>> can > >>>>>>>>>>> recommend or should I leave the null and allow it to fall back > to > >>>>>>> the > >>>>>>>>>>> KGroupedStream.name? > >>>>>>>>>>> > >>>>>>>>>>> Should this be expanded to handle grouped tables? This would be > >>>>>>>> pretty > >>>>>>>>>> easy > >>>>>>>>>>> for a normal aggregate but one allowing session stores and > >>> windowed > >>>>>>>>>> stores > >>>>>>>>>>> would required KTableSessionWindowAggregate and > >>>>>>> KTableWindowAggregate > >>>>>>>>>>> implementations. > >>>>>>>>>>> > >>>>>>>>>>> Thanks, > >>>>>>>>>>> Kyle > >>>>>>>>>>> > >>>>>>>>>>> On May 4, 2017 1:24 PM, "Eno Thereska" <eno.there...@gmail.com > > > >>>>>>>> wrote: > >>>>>>>>>>> > >>>>>>>>>>>> I’ll look as well asap, sorry, been swamped. > >>>>>>>>>>>> > >>>>>>>>>>>> Eno > >>>>>>>>>>>>> On May 4, 2017, at 6:17 PM, Damian Guy <damian@gmail.com > > > >>>>>>>> wrote: > >>>>>>>>>>>>> > >>>>>>>>>>>>> Hi Kyle, > >>>>>>>>>>>>> > >>>>>>>>>>>>> Thanks for the KIP. I apologize that i haven't had the chance > >>> to > >>>>>>>> look > >>>>>>>>>> at > >>>>>>>>>>>>> the KIP yet, but will schedule some time to look into it > >>>>>>> tomorrow. > >>>>>>>>> For > >>>>>>>>>>>> the > >>>>>>>>>>>>> implementation, can you raise a PR against kafka trunk and > mark > >>>>>>> it > >>>>>>>> as > >>>>>>>>>>>> WIP? > >>>>>>>>>>>>> It will be easier to review what you have done. > >>>>>>>>>>>>> > >>>>>>>>>>>>> Thanks, > >>>>>>>>>>>>> Damian > >>>>>>>>>>>>> > >>>>>>>>>>>>> On Thu, 4 May 2017 at 11:50 Kyle Winkelman < > >>>>>>>> winkelman.k...@gmail.com > >>>>>>>>>> > >>>>>>>>>>>> wrote: > >>>>>>>>>>>>> > >>>>>>>>>>>>>> I am replying to this in hopes it will draw some attention > to > >>> my > >>>>>>>> KIP > >>>>>>>>>> as > >>>>>>>>>>>> I > >>>>>>>>>>>>>> haven't heard from anyone in a couple days. This is my first > >>> KIP > >>>>>>>> and > >>>>>>>>>> my > >>>>>>>>>>>>>> first large contribution to the project so I'm sure I did > >>>>>>>> something > >>>>>>>>>>>> wrong. > >>>>>>>>>>>>>> ;) > >>>>>>>>>>>>>> > >>>>>>>>>>>>>> On May 1, 2017 4:18 PM, "Kyle Winkelman" < > >>>>>>>> winkelman.k...@gmail.com> > >>>>>>>>>>>> wrote: > >>>>>>>>>>>>>> > >>>>>>>>>>>>>>> Hello all, > >>>>>>>>>>>>>>> > >>>>>>>>>>>>>>> I have created KIP-150 to facilitate discussion about > adding > >>>>>>>>> cogroup > >>>>>>>>>> to > >>>>>>>>>>>>>>> the streams DSL. > >>>>>>>>>>>>>>> > >>>>>>>>>>>>>>> Please find the KIP here: > >>>>>>>>>>>>>>> https://cwiki.apache.org/confluence/display/KAFKA/KIP- > >>>>>>>>>>>>>>> 150+-+Kafka-Streams+Cogroup > >>>>>>>>>>>>>>> > >>>>>>>>>>>>>>> Please find my initial implementation here: > >>>>>>>>>>>>>>> https://github.com/KyleWinkelman/kafka > >>>>>>>>>>>>>>> > >>>>>>>>>>>>>>> Thanks, > >>>>>>>>>>>>>>> Kyle Winkelman > >>>>>>>>>>>>>>> > >>>>>>>>>>>>>> > >>>>>>>>>>>> > >>>>>>>>>>>> > >>>>>>>>>>> > >>>>>>>>>> > >>>>>>>>>> > >>>>>>>>> > >>>>>>>> > >>>>>>> > >>>>> > >>>>> > >>> > >>> > >

Re: [DISCUSS] KIP-150 - Kafka-Streams Cogroup

;> .cogroup(grouped3, aggregator3) >>>>>>>>> .aggregate(); >>>>>>>>> >>>>>>>>> As you can see this creates 1 StateStore, requires 1 initializer, >>> and >>>>> 1 >>>&

Re: [DISCUSS] KIP-150 - Kafka-Streams Cogroup

gh >> >>>> a >> >>>>>> KStreamAggreagte and grab the current aggregate from storeName1. It >> >>>> will >> >>>>>> add its incoming object to the aggregate, update the store and pass >> >> the >> >>>>>

Re: [DISCUSS] KIP-150 - Kafka-Streams Cogroup

> > >>>>>> wrote: > >>>>>> > >>>>>>> Kyle, > >>>>>>> > >>>>>>> thanks a lot for the KIP. Maybe I am a little slow, but I could not > >>>>>>> follow completely. Could you maybe add a more concrete example, > like > >>>> 3 > >>>>>>> streams with 3 records each (plus expected result), and show the > >>>>>>> difference between current way to to implement it and the proposed > >>>> API? > >>>>>>> This could also cover the internal processing to see what store > calls > >>>>>>> would be required for both approaches etc. > >>>>>>> > >>>>>>> I think, it's pretty advanced stuff you propose, and it would help > to > >>>>>>> understand it better. > >>>>>>> > >>>>>>> Thanks a lot! > >>>>>>> > >>>>>>> > >>>>>>> -Matthias > >>>>>>> > >>>>>>> > >>>>>>> > >>>>>>> On 5/4/17 11:39 AM, Kyle Winkelman wrote: > >>>>>>>> I have made a pull request. It can be found here. > >>>>>>>> > >>>>>>>> https://github.com/apache/kafka/pull/2975 > >>>>>>>> > >>>>>>>> I plan to write some more unit tests for my classes and get around > >>>> to > >>>>>>>> writing documentation for the public api additions. > >>>>>>>> > >>>>>>>> One thing I was curious about is during the > >>>>>>> KCogroupedStreamImpl#aggregate > >>>>>>>> method I pass null to the KGroupedStream#repartitionIfRequired > >>>>> method. > >>>>>> I > >>>>>>>> can't supply the store name because if more than one grouped > stream > >>>>>>>> repartitions an error is thrown. Is there some name that someone > >>>> can > >>>>>>>> recommend or should I leave the null and allow it to fall back to > >>>> the > >>>>>>>> KGroupedStream.name? > >>>>>>>> > >>>>>>>> Should this be expanded to handle grouped tables? This would be > >>>>> pretty > >>>>>>> easy > >>>>>>>> for a normal aggregate but one allowing session stores and > windowed > >>>>>>> stores > >>>>>>>> would required KTableSessionWindowAggregate and > >>>> KTableWindowAggregate > >>>>>>>> implementations. > >>>>>>>> > >>>>>>>> Thanks, > >>>>>>>> Kyle > >>>>>>>> > >>>>>>>> On May 4, 2017 1:24 PM, "Eno Thereska" <eno.there...@gmail.com> > >>>>> wrote: > >>>>>>>> > >>>>>>>>> I’ll look as well asap, sorry, been swamped. > >>>>>>>>> > >>>>>>>>> Eno > >>>>>>>>>> On May 4, 2017, at 6:17 PM, Damian Guy <damian@gmail.com> > >>>>> wrote: > >>>>>>>>>> > >>>>>>>>>> Hi Kyle, > >>>>>>>>>> > >>>>>>>>>> Thanks for the KIP. I apologize that i haven't had the chance to > >>>>> look > >>>>>>> at > >>>>>>>>>> the KIP yet, but will schedule some time to look into it > >>>> tomorrow. > >>>>>> For > >>>>>>>>> the > >>>>>>>>>> implementation, can you raise a PR against kafka trunk and mark > >>>> it > >>>>> as > >>>>>>>>> WIP? > >>>>>>>>>> It will be easier to review what you have done. > >>>>>>>>>> > >>>>>>>>>> Thanks, > >>>>>>>>>> Damian > >>>>>>>>>> > >>>>>>>>>> On Thu, 4 May 2017 at 11:50 Kyle Winkelman < > >>>>> winkelman.k...@gmail.com > >>>>>>> > >>>>>>>>> wrote: > >>>>>>>>>> > >>>>>>>>>>> I am replying to this in hopes it will draw some attention to > my > >>>>> KIP > >>>>>>> as > >>>>>>>>> I > >>>>>>>>>>> haven't heard from anyone in a couple days. This is my first > KIP > >>>>> and > >>>>>>> my > >>>>>>>>>>> first large contribution to the project so I'm sure I did > >>>>> something > >>>>>>>>> wrong. > >>>>>>>>>>> ;) > >>>>>>>>>>> > >>>>>>>>>>> On May 1, 2017 4:18 PM, "Kyle Winkelman" < > >>>>> winkelman.k...@gmail.com> > >>>>>>>>> wrote: > >>>>>>>>>>> > >>>>>>>>>>>> Hello all, > >>>>>>>>>>>> > >>>>>>>>>>>> I have created KIP-150 to facilitate discussion about adding > >>>>>> cogroup > >>>>>>> to > >>>>>>>>>>>> the streams DSL. > >>>>>>>>>>>> > >>>>>>>>>>>> Please find the KIP here: > >>>>>>>>>>>> https://cwiki.apache.org/confluence/display/KAFKA/KIP- > >>>>>>>>>>>> 150+-+Kafka-Streams+Cogroup > >>>>>>>>>>>> > >>>>>>>>>>>> Please find my initial implementation here: > >>>>>>>>>>>> https://github.com/KyleWinkelman/kafka > >>>>>>>>>>>> > >>>>>>>>>>>> Thanks, > >>>>>>>>>>>> Kyle Winkelman > >>>>>>>>>>>> > >>>>>>>>>>> > >>>>>>>>> > >>>>>>>>> > >>>>>>>> > >>>>>>> > >>>>>>> > >>>>>> > >>>>> > >>>> > >> > >> > >

Re: [DISCUSS] KIP-150 - Kafka-Streams Cogroup

; API? >>>>>>> This could also cover the internal processing to see what store calls >>>>>>> would be required for both approaches etc. >>>>>>> >>>>>>> I think, it's pretty advanced stuff you propose, and it

Re: [DISCUSS] KIP-150 - Kafka-Streams Cogroup

gt;>> method I pass null to the KGroupedStream#repartitionIfRequired >>> method. >>>> I >>>>>> can't supply the store name because if more than one grouped stream >>>>>> repartitions an error is thrown. Is there some name that some

Re: [DISCUSS] KIP-150 - Kafka-Streams Cogroup

s. > > > > > > > > > > Thanks, > > > > > Kyle > > > > > > > > > > On May 4, 2017 1:24 PM, "Eno Thereska" <eno.there...@gmail.com> > > wrote: > > > > > > > > > >> I’ll look as well asap, sorry, been swamped. > > > > >> > > > > >> Eno > > > > >>> On May 4, 2017, at 6:17 PM, Damian Guy <damian@gmail.com> > > wrote: > > > > >>> > > > > >>> Hi Kyle, > > > > >>> > > > > >>> Thanks for the KIP. I apologize that i haven't had the chance to > > look > > > > at > > > > >>> the KIP yet, but will schedule some time to look into it > tomorrow. > > > For > > > > >> the > > > > >>> implementation, can you raise a PR against kafka trunk and mark > it > > as > > > > >> WIP? > > > > >>> It will be easier to review what you have done. > > > > >>> > > > > >>> Thanks, > > > > >>> Damian > > > > >>> > > > > >>> On Thu, 4 May 2017 at 11:50 Kyle Winkelman < > > winkelman.k...@gmail.com > > > > > > > > >> wrote: > > > > >>> > > > > >>>> I am replying to this in hopes it will draw some attention to my > > KIP > > > > as > > > > >> I > > > > >>>> haven't heard from anyone in a couple days. This is my first KIP > > and > > > > my > > > > >>>> first large contribution to the project so I'm sure I did > > something > > > > >> wrong. > > > > >>>> ;) > > > > >>>> > > > > >>>> On May 1, 2017 4:18 PM, "Kyle Winkelman" < > > winkelman.k...@gmail.com> > > > > >> wrote: > > > > >>>> > > > > >>>>> Hello all, > > > > >>>>> > > > > >>>>> I have created KIP-150 to facilitate discussion about adding > > > cogroup > > > > to > > > > >>>>> the streams DSL. > > > > >>>>> > > > > >>>>> Please find the KIP here: > > > > >>>>> https://cwiki.apache.org/confluence/display/KAFKA/KIP- > > > > >>>>> 150+-+Kafka-Streams+Cogroup > > > > >>>>> > > > > >>>>> Please find my initial implementation here: > > > > >>>>> https://github.com/KyleWinkelman/kafka > > > > >>>>> > > > > >>>>> Thanks, > > > > >>>>> Kyle Winkelman > > > > >>>>> > > > > >>>> > > > > >> > > > > >> > > > > > > > > > > > > > > > > > > >

Re: [DISCUSS] KIP-150 - Kafka-Streams Cogroup

l.com> > wrote: > > > > > > > >> I’ll look as well asap, sorry, been swamped. > > > >> > > > >> Eno > > > >>> On May 4, 2017, at 6:17 PM, Damian Guy <damian@gmail.com> > wrote: > > > >>> > > > >>&g

Re: [DISCUSS] KIP-150 - Kafka-Streams Cogroup

o look into it tomorrow. > For > > >> the > > >>> implementation, can you raise a PR against kafka trunk and mark it as > > >> WIP? > > >>> It will be easier to review what you have done. > > >>> > > >>> Thanks, > > >>> Damian > > >>> > > >>> On Thu, 4 May 2017 at 11:50 Kyle Winkelman <winkelman.k...@gmail.com > > > > >> wrote: > > >>> > > >>>> I am replying to this in hopes it will draw some attention to my KIP > > as > > >> I > > >>>> haven't heard from anyone in a couple days. This is my first KIP and > > my > > >>>> first large contribution to the project so I'm sure I did something > > >> wrong. > > >>>> ;) > > >>>> > > >>>> On May 1, 2017 4:18 PM, "Kyle Winkelman" <winkelman.k...@gmail.com> > > >> wrote: > > >>>> > > >>>>> Hello all, > > >>>>> > > >>>>> I have created KIP-150 to facilitate discussion about adding > cogroup > > to > > >>>>> the streams DSL. > > >>>>> > > >>>>> Please find the KIP here: > > >>>>> https://cwiki.apache.org/confluence/display/KAFKA/KIP- > > >>>>> 150+-+Kafka-Streams+Cogroup > > >>>>> > > >>>>> Please find my initial implementation here: > > >>>>> https://github.com/KyleWinkelman/kafka > > >>>>> > > >>>>> Thanks, > > >>>>> Kyle Winkelman > > >>>>> > > >>>> > > >> > > >> > > > > > > > >

Re: [DISCUSS] KIP-150 - Kafka-Streams Cogroup

w what you have done. > >>> > >>> Thanks, > >>> Damian > >>> > >>> On Thu, 4 May 2017 at 11:50 Kyle Winkelman <winkelman.k...@gmail.com> > >> wrote: > >>> > >>>> I am replying to this in hopes it will draw some a

Re: [DISCUSS] KIP-150 - Kafka-Streams Cogroup

s in hopes it will draw some attention to my KIP as >> I >>>> haven't heard from anyone in a couple days. This is my first KIP and my >>>> first large contribution to the project so I'm sure I did something >> wrong. >>>> ;) >>>> >>&g

Re: [DISCUSS] KIP-150 - Kafka-Streams Cogroup

t; >> first large contribution to the project so I'm sure I did something > wrong. > >> ;) > >> > >> On May 1, 2017 4:18 PM, "Kyle Winkelman" <winkelman.k...@gmail.com> > wrote: > >> > >>> Hello all, > >>> > >>&g

[GitHub] kafka pull request #2975: KIP-150 [WIP]: Kafka Streams Cogroup

GitHub user KyleWinkelman opened a pull request: https://github.com/apache/kafka/pull/2975 KIP-150 [WIP]: Kafka Streams Cogroup Work in progress PR for KIP-150. You can merge this pull request into a Git repository by running: $ git pull https://github.com/KyleWinkelman/kafka

Re: [DISCUSS] KIP-150 - Kafka-Streams Cogroup

Winkelman" <winkelman.k...@gmail.com> wrote: >> >>> Hello all, >>> >>> I have created KIP-150 to facilitate discussion about adding cogroup to >>> the streams DSL. >>> >>> Please find the KIP here: >>> https:/

Re: [DISCUSS] KIP-150 - Kafka-Streams Cogroup

se find the KIP here: > > https://cwiki.apache.org/confluence/display/KAFKA/KIP- > > 150+-+Kafka-Streams+Cogroup > > > > Please find my initial implementation here: > > https://github.com/KyleWinkelman/kafka > > > > Thanks, > > Kyle Winkelman > > >

Re: [DISCUSS] KIP-150 - Kafka-Streams Cogroup

il.com> wrote: > Hello all, > > I have created KIP-150 to facilitate discussion about adding cogroup to > the streams DSL. > > Please find the KIP here: > https://cwiki.apache.org/confluence/display/KAFKA/KIP- > 150+-+Kafka-Streams+Cogroup > > Please find my ini

[DISCUSS] KIP-150 - Kafka-Streams Cogroup

Hello all, I have created KIP-150 to facilitate discussion about adding cogroup to the streams DSL. Please find the KIP here: https://cwiki.apache.org/confluence/display/KAFKA/KIP-150+-+Kafka-Streams+Cogroup Please find my initial implementation here: https://github.com/KyleWinkelman/kafka

Re: Kafka-Streams: Cogroup

Kyle, What's your apache id? I can grant you the permission. Guozhang On Sat, Apr 29, 2017 at 7:33 AM, Kyle Winkelman wrote: > I don't seem to have permission. When logged in I can neither edit the > main page nor create an additional KIP. > > Thanks, > Kyle > > On

Re: Kafka-Streams: Cogroup

I don't seem to have permission. When logged in I can neither edit the main page nor create an additional KIP. Thanks, Kyle On Thu, Apr 27, 2017 at 12:35 PM, Eno Thereska wrote: > Hi Kyle, > > I believe Guozhang has now given you permission to edit the KIP wiki at >

Re: Kafka-Streams: Cogroup

Hi Kyle, I believe Guozhang has now given you permission to edit the KIP wiki at https://cwiki.apache.org/confluence/display/KAFKA/Kafka+Improvement+Proposals. Could you see if you can add this there? Many thanks Eno On Wed, Apr 26, 2017 at 6:00 PM, Kyle Winkelman

Re: Kafka-Streams: Cogroup

ngle store that is >> used in all of the processors. >> >> Please let me know if this is something you think would add value to >> kafka streams. And I will try to create a KIP to foster more communication. >> >> You can take a look at what I have. I think it's missing a

Re: Kafka-Streams: Cogroup

Hi Kyle, Sorry for the delay in replying. I think it's worth doing a KIP for this one. One super helpful thing with KIPs is to list a few more scenarios that would benefit from this approach. In particular it seems the main benefit is from reducing the number of state stores. Does this

Re: Kafka-Streams: Cogroup

Eno, Thanks for the response. The figure was just a restatement of my questions. I have made an attempt at a low level processor and it appears to work but it isn't very pretty and was hoping for something at the streams api level. I have written some code to show an example of how I see the

  1   2   >