GitHub user jkff opened a pull request:
https://github.com/apache/beam/pull/2455
[BEAM-1903, BEAM-1904] Fixes SDF issues re: watermarks and stop/resume
See detailed discussion in document:
https://docs.google.com/document/d/1BGc8pM1GOvZhwR9SARSVte-20XEoBUxrGJ5gTWXdv3c/edit
In particular, this PR:
- Removes DoFn.ProcessContinuation and all the code and tests that handled
returning resume() - from DoFnSignatures, DoFnInvokers, and SplittableParDo and
runner hooks, as well as inference of UnboundedPerElement from the return value.
- Adds ProcessContext.updateWatermark(), currently allowed to be called
only from SDF. This has surprisingly little impact - basically just a "throw
unsupported" implementation in SimpleDoFnRunner and DoFnTester, and a proper
implementation in SDF code
- Adds RestrictionTracker.checkDone() and unit tests
Overall this looks much easier of a change than I thought, and should be
pretty easy to review.
R: @tgroh
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/jkff/incubator-beam sdf-updates
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/beam/pull/2455.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #2455
----
commit 77260e65f7bbf8e9e6e8043f4a22848639843ddb
Author: Eugene Kirpichov <[email protected]>
Date: 2017-04-06T22:07:05Z
Fixes SDF issues re: watermarks and stop/resume
See detailed discussion in document:
https://docs.google.com/document/d/1BGc8pM1GOvZhwR9SARSVte-20XEoBUxrGJ5gTWXdv3c/edit
----
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [email protected] or file a JIRA ticket
with INFRA.
---
