Re: Firing windows multiple times

aggregating over a 30 day window: if it filters out early fires, wouldn't it only produce new/unique results every 30 days? I very well may have misunderstood this solution. -- View this message in context: http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/Firing-windows-multiple-times

Re: Firing windows multiple times

up to speed? > > Thank you much! > > > > -- > View this message in context: > http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/Firing-windows-multiple-times-tp8424p8975.html > Sent from the Apache Flink User Mailing List archive. mailing list archive > at Nabble.com. >

Re: Firing windows multiple times

sing guarantees > when a failure/recovery occurs if we don't do it carefully. Also, we're not > particularly sophisticated yet with regard to avoiding unnecessary queries > to the time series data. > > -Shannon > > > From: Aljoscha Krettek <aljos...@apache.org> &g

Re: Firing windows multiple times

k <aljos...@apache.org<mailto:aljos...@apache.org>> Date: Friday, September 2, 2016 at 4:02 AM To: "user@flink.apache.org<mailto:user@flink.apache.org>" <user@flink.apache.org<mailto:user@flink.apache.org>> Subject: Re: Firing windows multiple times I see, I didn'

Re: Firing windows multiple times

verage 0.5 months stale. A year-long window is even worse. > > -Shannon > > From: Aljoscha Krettek <aljos...@apache.org> > Date: Tuesday, August 30, 2016 at 9:08 AM > To: Shannon Carey <sca...@expedia.com>, "user@flink.apache.org" < > user@flink.apache

Re: Firing windows multiple times

n Carey <sca...@expedia.com<mailto:sca...@expedia.com>>, "user@flink.apache.org<mailto:user@flink.apache.org>" <user@flink.apache.org<mailto:user@flink.apache.org>> Subject: Re: Firing windows multiple times Hi, I think this can be neatly expressed by using somet

Re: Firing windows multiple times

ache.org>>, "user@flink.apache.org<mailto:user@flink.apache.org>" <user@flink.apache.org<mailto:user@flink.apache.org>> Subject: Re: Firing windows multiple times Thanks Aljoscha, I didn't know about those. Yes, they look like handy changes, especially to enable fle

Re: Firing windows multiple times

e.org<mailto:user@flink.apache.org>" <user@flink.apache.org<mailto:user@flink.apache.org>> Subject: Re: Firing windows multiple times Hi, there is already this FLIP: https://cwiki.apache.org/confluence/display/FLINK/FLIP-4+%3A+Enhance+Window+Evictor which also links to a mailin

Re: Firing windows multiple times

Hi, there is already this FLIP: https://cwiki.apache.org/confluence/display/FLINK/FLIP-4+%3A+Enhance+Window+Evictor which also links to a mailing list discussion. And this FLIP: https://cwiki.apache.org/confluence/display/FLINK/FLIP-2+Extending+Window+Function+Metadata. The former proposes to

Re: Firing windows multiple times

"If Window B is a Folding Window and does not have an evictor then it should not keep the list of all received elements." Agreed! Upon closer inspection, the behavior I'm describing is only present when using EvictingWindowOperator, not when using WindowOperator. I misread line 382 of

Re: Firing windows multiple times

t=2), but internal >Window state looks like *[x(time=1, count=1), y(time=1, count=2), > z(time=1, count=2)]* > > As you can see, the internal window state continues to grow despite what > fold() is doing. > > Does that explanation help interpret my original email? > &g

Re: Firing windows multiple times

> purges and emits current state as event z(time=1, count=2) >> Window B receives event, trigger waits for processing time delay, then >> executes fold() and emits event(time=1 => count=2), but internal Window >> state looks like [x(time=1, count=1), y(time=1, count=2), z(time=1, coun

Re: Firing windows multiple times

tion help interpret my original email? > > -Shannon > > > From: Aljoscha Krettek <aljos...@apache.org <mailto:aljos...@apache.org>> > Date: Wednesday, August 10, 2016 at 12:18 PM > To: "user@flink.apache.org <mailto:user@flink.apache.org>" > <us

Re: Firing windows multiple times

From: Aljoscha Krettek <aljos...@apache.org<mailto:aljos...@apache.org>> Date: Wednesday, August 10, 2016 at 12:18 PM To: "user@flink.apache.org<mailto:user@flink.apache.org>" <user@flink.apache.org<mailto:user@flink.apache.org>> Subject: Re:

Re: Firing windows multiple times

Hi Aljoscha, This looks like the bug that we discussed, as part of Enhance window evictor JIRA Thanks, Vishnu On Wed, Aug 10, 2016 at 1:18 PM, Aljoscha Krettek wrote: > Hi, > from your mail I'm gathering that you are in fact using an Evictor, is > that correct? If not,

Re: Firing windows multiple times

Hi, from your mail I'm gathering that you are in fact using an Evictor, is that correct? If not, then the window operator should not keep all the elements ever received for a window but only the aggregated result. Side note, there seems to be a bug in EvictingWindowOperator that causes evicted

Re: Firing windows multiple times

One unfortunate aspect of using a fold() instead of a window is that the fold function has no knowledge of the watermarks. As a result, it is difficult to ensure that only items before the current watermark are included in the aggregation, and that old items are evicted correctly. This fact

Firing windows multiple times

I recently noticed something about windows: they retain (in state) every element that they receive regardless of whether the user provides a fold/reduce function. I can tell that such an approach is necessary in order for evictors to work, but I'm not sure if there are other reasons. I'll