Minbo Bae created BEAM-12751:
--------------------------------
Summary: Python DataflowRunner doesn't set client_request_id
Key: BEAM-12751
URL: https://issues.apache.org/jira/browse/BEAM-12751
Project: Beam
Issue Type: Bug
Components: runner-dataflow
Environment: python
Reporter: Minbo Bae
Currently, [Java DataflowRunner|#L1026] sets {{clientRequestId}}, but [Python
DataflowRunner|https://github.com/apache/beam/blob/v2.31.0/sdks/python/apache_beam/runners/dataflow/dataflow_runner.py]
has no such logic.
This causes an issue with job creation when the first try is failed with
DEADLINE_EXCEEDED (but actually job successfully created), because the second
try will fail with "There is already an active job named...". As per [Dataflow
public
doc|https://cloud.google.com/dataflow/docs/reference/rest/v1b3/projects.locations.jobs],
`clientRequestId` must be set to avoid this issue.
In addition, as the retry happens in
[google-apitools|https://github.com/google/apitools/blob/v0.5.31/apitools/base/py/http_wrapper.py#L346-L360]
and it doesn't print any logs, the error message is difficult to understand
and there is actually a successfully created job even if it returns the error.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
