Chad Dombrova created BEAM-8539:
-----------------------------------
Summary: Clearly define the valid job state transitions
Key: BEAM-8539
URL: https://issues.apache.org/jira/browse/BEAM-8539
Project: Beam
Issue Type: Improvement
Components: beam-model, runner-core, sdk-java-core, sdk-py-core
Reporter: Chad Dombrova
The Beam job state transitions are ill-defined, which is big problem for
anything that relies on the values coming from JobAPI.GetStateStream.
I was hoping to find something like a state transition diagram in the docs so
that I could determine the start state, the terminal states, and the valid
transitions, but I could not find this. The code reveals that the SDKs differ
on the fundamentals:
Java InMemoryJobService:
* start state (prepared): STOPPED
* about to submit to executor: STARTING
* actually running on executor: RUNNING
* terminal states: DONE, FAILED, CANCELLED, DRAINED
Python AbstractJobServiceServicer / LocalJobServicer:
* start state: STARTING
* terminal states: DONE, FAILED, CANCELLED, STOPPED
I think it would be good to make python work like Java, so that there is a
difference in state between a job that has been prepared and one that has been
run.
It's hard to tell how far this problem has spread within the various runners.
I think a simple thing that can be done to help standardize behavior is to
implement the terminal states as an enum in the beam_job_api.proto, or create a
utility function in each language for checking if a state is terminal, so that
it's not left up to each runner to reimplement this logic.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
