Thanks Jon for bringing this up. We have seen the community discussing about explicit triggers for Kafka Streams, and some of them have been covered in KIP-63:
https://cwiki.apache.org/confluence/display/KAFKA/KIP-63%3A+Unify+store+and+downstream+caching+in+streams Guozhang On Thu, Dec 1, 2016 at 8:43 AM, Jon Yeargers <jon.yearg...@cedexis.com> wrote: > Seems like there are many questions on SO and related about "how do I know > when my windowed aggregation is 'done'?" The answer has always been "It's > never done. You're thinking about it the wrong way." > > I propose a new function for KStream: > > finiteAggregateByKey(Initializer, Aggregator, Windows, Timeout) > > where 'Timeout' would be a struct of how many seconds after the window is > finished and a topic to dump the resultant values to. > > EG I have a topic 'bar' with an 'rolling' aggregator of 20 minutes / 1 > minute. During the 20 minutes it's building a list of things from the 'bar' > topic. I tell it to terminate 5 minutes after the initial 20 have elapsed > and write out the final values from 'T > <https://kafka.apache.org/0100/javadoc/org/apache/kafka/ > streams/kstream/Aggregator.html> > aggregate' to a topic 'allBar'. This would allow me to put a consumer on > that topic and watch for new entries. > > Subsequent values that would have matched said aggregator-window above are > ignored. > -- -- Guozhang
