Re: kafka-streams / window expiration because of shuffling

2020-11-23 Thread Mathieu D
Hi John Thanks for your message. Indeed there's a potential feature here. Should I log it on Jira ? Mathieu Le ven. 20 nov. 2020 à 21:53, John Roesler a écrit : > Hi Mathieu, > > Ah, that is unfortunate. I believe your analysis is correct. In general, > we have no good solution to the problem

Re: kafka-streams / window expiration because of shuffling

2020-11-20 Thread John Roesler
Hi Mathieu, Ah, that is unfortunate. I believe your analysis is correct. In general, we have no good solution to the problem of upstream tasks moving ahead of each other and causing disorder in the repartition topics. Guozhang has done a substantial amount of thinking on this subject, though,

kafka-streams / window expiration because of shuffling

2020-11-20 Thread Mathieu D
Hello there, We're processing IOT device data, each device sending several metrics. When we upgrade our streams app, we set a brand new 'application.id' to reprocess a bit of past data, to warm up the state stores and aggregations to make sure all outputs will be valid. Downstream is designed