[
https://issues.apache.org/jira/browse/BEAM-241?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Kenneth Knowles reassigned BEAM-241:
------------------------------------
Assignee: Kenneth Knowles (was: Frances Perry)
> Not easy for runners to get late-data dropping
> ----------------------------------------------
>
> Key: BEAM-241
> URL: https://issues.apache.org/jira/browse/BEAM-241
> Project: Beam
> Issue Type: Bug
> Components: runner-core
> Reporter: Mark Shields
> Assignee: Kenneth Knowles
>
> Quite by accident realized the Flink runner delegates to
> GroupAlsoByWindowViaWindowSetDoFn for GBK, which in turn delegates to
> ReduceFnRunner. The latter now assumes no messages will arrive beyond the
> 'garbage collection' time of their target window(s).
> The Dataflow runner interposes a LateDataDroppingDoFnRunner into the path so
> as to drop those too-late messages. That's done (I think) using
> DoFnRunners.createDefault.
> I don't think the Flink runner does that.
> We need a nice runner-friendly way of dealing with the too-late data.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
