[
https://issues.apache.org/jira/browse/BEAM-2447?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16052430#comment-16052430
]
ASF GitHub Bot commented on BEAM-2447:
--------------------------------------
GitHub user jkff opened a pull request:
https://github.com/apache/beam/pull/3383
[BEAM-2447] Reintroduces DoFn.ProcessContinuation (Dataflow worker
compatibility part)
Prerequisite for #3360. Will need a Dataflow worker update too.
R: @tgroh
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/jkff/incubator-beam process-cont-base
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/beam/pull/3383.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #3383
----
commit 71599e1a7f0766ac5e7ea100f8bbc69152e44860
Author: Eugene Kirpichov <[email protected]>
Date: 2017-06-16T21:56:07Z
Reintroduces DoFn.ProcessContinuation (Dataflow worker compatibility part)
----
> Reintroduce DoFn.ProcessContinuation
> ------------------------------------
>
> Key: BEAM-2447
> URL: https://issues.apache.org/jira/browse/BEAM-2447
> Project: Beam
> Issue Type: Bug
> Components: sdk-java-core
> Reporter: Eugene Kirpichov
> Assignee: Eugene Kirpichov
>
> ProcessContinuation.resume() is useful for tailing files - when we reach
> current EOF, we want to voluntarily suspend the process() call rather than
> wait for runner to checkpoint us.
> In BEAM-1903, DoFn.ProcessContinuation was removed because there was
> ambiguity about the semantics of resume() especially w.r.t. the following
> situation described in
> https://docs.google.com/document/d/1BGc8pM1GOvZhwR9SARSVte-20XEoBUxrGJ5gTWXdv3c/edit
> : the runner has taken a checkpoint on the tracker, and then the
> ProcessElement call returns resume() signaling that the work is still not
> done - then there's 2 checkpoints to deal with.
> Instead, the proper way to refine this semantics is:
> - After checkpoint() on a RestrictionTracker, the tracker MUST fail all
> subsequent tryClaim() calls, and MUST succeed in checkDone().
> - After a failed tryClaim() call, the ProcessElement method MUST return stop()
> - So ProcessElement can return resume() only *instead* of doing tryClaim()
> - Then, if the runner has already taken a checkpoint but tracker has returned
> resume(), we do not need to take a new checkpoint - the one already taken
> already accurately describes the remainder of the work.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
