pmcquighan-camus commented on issue #57359:
URL: https://github.com/apache/airflow/issues/57359#issuecomment-3848772245
Hi - we have been using `apache-airflow-providers-google==19.3.0` for the
past several weeks and the issue recurred:
```
[2026-02-02 04:42:52] ERROR - Exception occurred while checking for job
completion.
source=airflow.providers.google.cloud.triggers.dataflow.TemplateJobStartTrigger
loc=dataflow.py:149
ServiceUnavailable: 503 Visibility check was unavailable. Please retry the
request and contact support if the problem persists
File
"/home/airflow/.local/lib/python3.12/site-packages/airflow/providers/google/cloud/triggers/dataflow.py",
line 113 in run
File
"/home/airflow/.local/lib/python3.12/site-packages/airflow/providers/google/cloud/hooks/dataflow.py",
line 1480 in get_job_status
File
"/home/airflow/.local/lib/python3.12/site-packages/airflow/providers/google/cloud/hooks/dataflow.py",
line 1457 in get_job
File
"/home/airflow/.local/lib/python3.12/site-packages/google/cloud/dataflow_v1beta3/services/jobs_v1_beta3/async_client.py",
line 478 in get_job
File
"/home/airflow/.local/lib/python3.12/site-packages/google/api_core/grpc_helpers_async.py",
line 88 in __await__
AioRpcError: <AioRpcError of RPC that terminated with:
status = StatusCode.UNAVAILABLE
details = "Visibility check was unavailable. Please retry the request
and contact support if the problem persists"
debug_error_string = "UNKNOWN:Error received from peer
ipv4:74.125.126.95:443 {created_time:"2026-02-02T12:42:52.166480427+00:00",
grpc_status:14, grpc_message:"Visibility check was unavailable. Please retry
the request and contact support if the problem persists"}"
>
File
"/home/airflow/.local/lib/python3.12/site-packages/google/api_core/grpc_helpers_async.py",
line 85 in __await__
File
"/home/airflow/.local/lib/python3.12/site-packages/grpc/aio/_interceptor.py",
line 472 in __await__
File "/home/airflow/.local/lib/python3.12/site-packages/grpc/aio/_call.py",
line 327 in __await__
```
The task retry error from the initial bug report `ValueError: dictionary
update sequence element #0 has length 1; 2 is required` no longer occurs but as
expected the task is still just considered failed since the triggerer marked
the task as failed, and the dataflow job is *not* actually retried (nor does
the triggerer or any component attempt to get the status a second time)
```
[2026-02-02 04:45:10] ERROR - Task failed with exception source=task
loc=task_runner.py:972
AirflowException: 503 Visibility check was unavailable. Please retry the
request and contact support if the problem persists
File
"/home/airflow/.local/lib/python3.12/site-packages/airflow/sdk/execution_time/task_runner.py",
line 920 in run
File
"/home/airflow/.local/lib/python3.12/site-packages/airflow/sdk/execution_time/task_runner.py",
line 1307 in _execute_task
File
"/home/airflow/.local/lib/python3.12/site-packages/airflow/sdk/bases/operator.py",
line 1632 in resume_execution
File
"/home/airflow/.local/lib/python3.12/site-packages/airflow/providers/google/cloud/operators/dataflow.py",
line 650 in execute_complete
```
> manage to find how to reproduce it, please comment (without clear
reproduction steps there's not too much that we can do).
it is dependent on the GCP dataflow api returning a 503 - the only way to
reliably reproduce it would be with a mock http server that the gcp client
connects to and having it return a 503 (or by mocking out the
`JobsV1Beta3AsyncClient.get_job` method to throw the AioRpcError in the
stacktrace above). i don't have permissions to reopen this issue but it is
definitely a still a bug with the retry logic in the provider
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
