Eugene Kirpichov created BEAM-1784:
--------------------------------------
Summary: DataflowPipelineJob.cancel() should be idempotent
Key: BEAM-1784
URL: https://issues.apache.org/jira/browse/BEAM-1784
Project: Beam
Issue Type: Bug
Components: runner-dataflow
Reporter: Eugene Kirpichov
Assignee: Eugene Kirpichov
Right now DataflowPipelineJob.cancel() issues an RPC to the Dataflow service
and, if it fails, checks whether the job is in a terminal state - in that case
the failure is ignored.
Ideally, the Dataflow-side cancel call itself would be idempotent, and would
succeed even if the job is already terminated. However, right now this is not
the case.
Additionally, Dataflow service can, for a short time, return RUNNING state even
for a job that has already been cancelled (i.e. it will reject cancel() calls,
but will report state as RUNNING).
Because of this, if you call DataflowPipelineJob.cancel() multiple times, it
might fail (throw an exception). Calling cancel() multiple times should,
logically, be ok. So, to partially work around these issues of the Dataflow
service, we can make DataflowPipelineJob.cancel() be idempotent by itself, i.e.
absorb duplicate cancel() requests. This, of course, doesn't address the case
when the job terminates externally almost concurrently to calling
DataflowPipelineJob.cancel() - but it's an improvement.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
