Good to know that the basic capability is in place, otherwise stateful processing could only be used with timers that perform cleanup in user land.
I don't think the cleanup timer is used in the portable Flink runner though. DoFnOperator.createWrappingDoFnRunner isn't executed in this case. Would be nice to have test coverage for the cleanup path, since that would eventually lead to out of memory (with the heap memory state backend) or even harder to diagnose disk space issues. Thomas On Wed, Mar 20, 2019 at 7:03 AM Maximilian Michels <[email protected]> wrote: > Thanks for the pointer Reuven. I didn't see that on window expiration > this would iterate over all user state and call the `clear` method. > > -Max > > On 20.03.19 14:59, Reuven Lax wrote: > > Is this not already handled by cleanupTimer in StatefulDoFnRunner? > > > > On Wed, Mar 20, 2019 at 6:23 AM Maximilian Michels <[email protected] > > <mailto:[email protected]>> wrote: > > > > Hi, > > > > I just realized that user state acquired via StateInternals in the > > Flink > > Runner is not released automatically even when it falls out of the > > Window scope. There are ways to work around this, i.e. setting a > > cleanup > > timer that fires when the Window expires. > > > > Do we expect Runners to perform the cleanup? I would think so since > > users do not have control over state once the window expires. > > > > Thanks, > > Max > > >
