Re: KafkaStreams GroupBy with new key. Can I skip repartition?

2020-03-02 Thread John Roesler
Hi, all, Sorry for the confusion. I didn’t look too closely at it, I was just going by the fact that it was listed under the scope of KIP-221. I agree that the final design of the KIP doesn’t have too much to do with the description of KAFKA-4835. Maybe we should remove that ticket from the

Re: KafkaStreams GroupBy with new key. Can I skip repartition?

2020-03-01 Thread Murilo Tavares
I agree with Mathias. Can’t see how this KIP/PR helps with the problem described in the KAFKA-4835... On Sun, Mar 1, 2020 at 2:16 PM Matthias J. Sax wrote: > -BEGIN PGP SIGNED MESSAGE- > Hash: SHA512 > > I don't think that KIP-221 addressed the discussed use case. > > KIP-221 allows to

Re: KafkaStreams GroupBy with new key. Can I skip repartition?

2020-03-01 Thread Matthias J. Sax
-BEGIN PGP SIGNED MESSAGE- Hash: SHA512 I don't think that KIP-221 addressed the discussed use case. KIP-221 allows to force a repartitioning manually, while the use case describe in the original email was to suppress/skip a repartitioning ste p. The issue to avoid unnecessary

Re: KafkaStreams GroupBy with new key. Can I skip repartition?

2020-02-29 Thread John Roesler
Hi all, The KIP is accepted and implemented already, but is blocked on code review: https://github.com/apache/kafka/pull/7170 A quick note on the lack of recent progress... It's completely our fault, the reviews fell by the wayside during the 2.5.0 release cycle, and we haven't gotten back to

Re: KafkaStreams GroupBy with new key. Can I skip repartition?

2020-02-29 Thread Guozhang Wang
It is in progress, but I was not the main reviewer of that ticket so I cannot say for sure. I saw the last update is on Jan/2019 so maybe it's a bit loose now.. If you want to pick it up and revive the KIP completion feel free to do so :) Guozhang On Fri, Feb 28, 2020 at 5:54 PM Murilo Tavares

Re: KafkaStreams GroupBy with new key. Can I skip repartition?

2020-02-28 Thread Murilo Tavares
Guozhang The ticket definitely describes what I’m trying to achieve. And should I be hopeful with the fact it’s in progress? :) Thanks for pointing that out. Murilo On Fri, Feb 28, 2020 at 2:57 PM Guozhang Wang wrote: > Hi Murilo, > > Would this be helping your case? >

Re: KafkaStreams GroupBy with new key. Can I skip repartition?

2020-02-28 Thread Guozhang Wang
Hi Murilo, Would this be helping your case? https://issues.apache.org/jira/browse/KAFKA-4835 Guozhang On Fri, Feb 28, 2020 at 7:01 AM Murilo Tavares wrote: > Hi > I am currently doing a simple KTable groupby().aggregate() in KafkaStreams. > In the groupBy I do need to select a new key, but I

KafkaStreams GroupBy with new key. Can I skip repartition?

2020-02-28 Thread Murilo Tavares
Hi I am currently doing a simple KTable groupby().aggregate() in KafkaStreams. In the groupBy I do need to select a new key, but I know for sure that the new key would still fall in the same partition. Because of this, I believe the repartition would not be necessary, but my question is: is it