Hi, you are right, this is a problem. In an earlier version we were only setting very few actual timers using the RuntimeContext because a firing timer will trigger all the timers with a lower timestamp that we have stored in the trigger queue. We have to change the lower level trigger service (in StreamTask) to only store one timer per very short time window, so that if the window operator registers thousands of timers for, say, time 15:30:03 it actually only saves one timer.
I created a Jira Issue: https://issues.apache.org/jira/browse/FLINK-3669 Cheers, Aljoscha On Thu, 24 Mar 2016 at 11:30 Konstantin Knauf <konstantin.kn...@tngtech.com> wrote: > Hi everyone, > > we were testing a Flink streaming job (1.0.0), with a GlobalWindow on a > KeyedStream with custom Trigger. > > On each element the trigger function registers a processing time timer > and deletes the currently registered processing time timer. So we are > registering a lot of timers, but also deleting most of them right away. > > The desired functionality is, that the window is purged (and all state > is set to null) after a timeout (last event for this key + timeout). > > The performance tests showed, that after a short time (5mins or so) all > the time went to garbage collection. From the heap dumnps, we can tell > that the problem were retained TriggerTasks (with reference to the > TriggerContext) off all the registered processing time timers. > > The problems seems to be that when deleting the TriggerTasks the > corresponding Callables are not removed form the queue, the > deleteProcessingTimeTimer-method only removes the Timer from the > set/queues of the TriggerContext itself, but not from the RuntimeContext. > > Is this a bug? Are we using ProcessingTimeTimers in a fundamentally > wrong way? If so, is there any other way to achieve the desired > functionality? > > We have a workaround in place now (basically just a timeout starting > with the first element in window instead of the last element in the > window). > > Cheers, > > Konstantin > > -- > Konstantin Knauf * konstantin.kn...@tngtech.com * +49-174-3413182 > TNG Technology Consulting GmbH, Betastr. 13a, 85774 Unterföhring > Geschäftsführer: Henrik Klagges, Christoph Stock, Dr. Robert Dahlke > Sitz: Unterföhring * Amtsgericht München * HRB 135082 >
