Re: Kafka Streams - issues with windowing and suppress

PR submitted :) https://github.com/apache/kafka/pull/8520 On Mon, Apr 20, 2020 at 2:34 PM John Roesler wrote: > Hi Liam, > > That sounds like a good idea to me. In fact, I’d go so far as to say we > should just change the existing example to include a grace period, and not > bother with an

Re: Unexpected behaviour on windowing aggregations

Yep, it sure was, now that I've removed the PEBKAC, numbers match up perfectly :) (A somewhat sheepish) thank you, Liam On Mon, 20 Apr. 2020, 2:51 pm John Roesler, wrote: > Hey Liam, > > Hah! Tell me about it... > > Well, let’s hope that was it. > Thanks, > John > > On Sun, Apr 19, 2020, at

Re: Unexpected behaviour on windowing aggregations

Hey Liam, Hah! Tell me about it... Well, let’s hope that was it. Thanks, John On Sun, Apr 19, 2020, at 18:43, Liam Clarke wrote: > Hi John, > > Thanks for the reply - yep, that was a dumb copy and paste error, which is > what I get for coding while surrounded by kids. >_< I'm deploying a

Re: Kafka Streams - issues with windowing and suppress

Hi Liam, That sounds like a good idea to me. In fact, I’d go so far as to say we should just change the existing example to include a grace period, and not bother with an extra example. That would put it front and center. A PR would be greatly appreciated! Thanks for the offer! Thanks, John

Re: Kafka Streams - issues with windowing and suppress

Hi Matthias, I think as an interim measure, if the windowing samples in the docs showed an additional example where the grace period was set (with perhaps a comment about the current default grace period, and planned future changes?) it would make it sufficiently visible - happy to submit a PR

Re: Kafka Streams - issues with windowing and suppress

I would prefer to not make the grace-period a mandatory argument and keep the API as-is. I understand the issue of backward compatibility, but I would still argue that we should just change the default grace period to 0 in the 3.0 release. It's a major release and thus it seems to be fine. To

Re: Unexpected behaviour on windowing aggregations

Hi John, Thanks for the reply - yep, that was a dumb copy and paste error, which is what I get for coding while surrounded by kids. >_< I'm deploying a fixed version of it as we speak. Thanks for the reply though :) Kind regards, Liam Clarke On Mon, 20 Apr. 2020, 2:08 am John Roesler,

Re: Need advice on how to deploy and update a streams app

Hi Mathieu, We've been deploying stateful Kafka streams apps in Kubernetes autoscaling on CPU and lag successfully for about a year now. We also automate releases of new versions of apps using Flux. Kafka Streams is really good at doing the right thing - e.g., it uses Kafka's consumer groups to

Need advice on how to deploy and update a streams app

Hey Kafka lovers, I have a lot of questioning around the roll-out and the update of kafka streams app (continuous deployment fashion). We are used to ship our apps in dockers quite frequently (our current orchestrator is AWS ECS). In the context of streams app, i understand we should be much

Re: Kafka Streams - issues with windowing and suppress

Oh, man, that’s a good idea. I can propose to deprecate (not remove) the existing ‘of’ factory method and add one with a mandatory grace period. Not sure why I didn’t think of that before. Probably too caught up in looking for something “smart”. Thanks! John On Sun, Apr 19, 2020, at 02:27,

Re: Unexpected behaviour on windowing aggregations

Hi Liam, I took a quick look. On the output side, it looks like you’re adding the count to the prior count. Should that just set the outbound vale to the new count? Maybe I misunderstood the situation. What I mean is, suppose you get two events for the same window: Inbound map := 0+1 = 1

Re: Retention period for __consumer_offsets topic

Hi Nitin, Default in Kafka 2.0+ is 7 days, previously it was 24 hours IIRC. Only reason you need to change it is if you anticipate having a whole consumer group offline for longer than your current retention period for debugging/maintenance etc. Cheers, Liam Clarke-Hutchinson On Sun, 19 Apr.

Re: Retention period for __consumer_offsets topic

**BUMP** Thanks, Nitin On Thu, Apr 16, 2020 at 6:51 PM nitin agarwal wrote: > Hi, > > What should be the retention period for __consumer_offsets topic? Should > it be the same as other Kafka topics? > > Thanks, > Nitin >

Unexpected behaviour on windowing aggregations

Hello all, I have been running this code against production data, and I'm emitting counts/sums for a sentinel record id to stdout so I can observe the behaviour: https://gist.github.com/LiamClarkeNZ/b101ce6a42a2e5e1efddfe3a98c5805f When this code is run, the window duration is 2 minutes, grace

Re: Kafka Streams - issues with windowing and suppress

Hi John, I can't really think of a way to make it more obvious without breaking backwards compatibility - e.g., obvious easy fix is that grace period is a mandatory arg to TimeWindows, but that would definitely break compatibility. Cheers, Liam Clarke-Hutchinson On Thu, Apr 16, 2020 at 1:59 AM