GitHub user jkff opened a pull request:
https://github.com/apache/incubator-beam/pull/1353
Liberates ReduceFnRunner from WindowingInternals, and lets
WindowingInternals do windowed side outputs
- Introduces WindowingInternals.sideOutputWindowedValue (will be necessary
for Splittable DoFn)
- Implements it properly in all runners (required some minor refactoring in
Spark and Flink ProcessContext implementations)
- Introduces "OutputWindowedValue" interface and "SideInputAccess"
interfaces, and uses them in ReduceFnRunner instead of directly using
WindowingInternals.
- Introduces adapters from WindowingInternals to these two interfaces, for
use in OldDoFn contexts
- Moves some StateContext functions into ReduceFnContextFactory, because
they make more sense in runners-core than in sdk (because they are only invoked
by different runners). The only remaining StateContexts function is
nullContext, but I couldn't figure out an easy way to move it into runners-core
and gave up (however in fact I'm not sure its current usages are correct at
all...)
R: @kennknowles (for bulk of the code and as committer)
CC: @aljoscha @amitsela (for the minor refactorings in respective runners)
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/jkff/incubator-beam
reducefn-windowing-internals
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/incubator-beam/pull/1353.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #1353
----
commit 14dd49e02ab2eabb7c7e42cb1e9cab29f7cea44e
Author: Eugene Kirpichov <[email protected]>
Date: 2016-11-11T02:40:53Z
Liberates ReduceFnRunner from WindowingInternals
commit d6c62454bb822063c863d2bcd1f2436453b94864
Author: Eugene Kirpichov <[email protected]>
Date: 2016-11-12T00:37:42Z
Refactor FlinkProcessContext more cleanly into single- and multi-output
versions
commit 14e5847f738b3d89808d5d612684fcddffd51b7f
Author: Eugene Kirpichov <[email protected]>
Date: 2016-11-12T00:57:27Z
Refactor SparkProcessContext more cleanly into single- and multi-output
versions
----
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [email protected] or file a JIRA ticket
with INFRA.
---
