Reuven Lax created BEAM-5040:
--------------------------------
Summary: BigQueryIO retries infinitely in WriteTable and
WriteRename
Key: BEAM-5040
URL: https://issues.apache.org/jira/browse/BEAM-5040
Project: Beam
Issue Type: Bug
Components: io-java-gcp
Affects Versions: 2.5.0
Reporter: Reuven Lax
Assignee: Reuven Lax
BigQueryIO retries infinitely in WriteTable and WriteRename
Several failure scenarios with the current code:
# It's possible for a load job to return failure even though it actually
succeeded (e.g. the reply might have timed out). In this case, BigQueryIO will
retry the job which will fail again (because the job id has already been used),
leading to indefinite retries. Correct behavior is to stop retrying as the load
job has succeeded.
# It's possible for a load job to be accepted by BigQuery, but then to fail on
the BigQuery side. In this case a retry with the same job id will fail as that
job id has already been used. BigQueryIO will sometimes detect this, but if the
worker has restarted it will instead issue a load with the old job id and go
into a retry loop. Correct behavior is to generate a new deterministic job id
and retry using that new job id.
# In many cases of worker restart, BigQueryIO ends up in infinite retry loops.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
