its-amann commented on code in PR #77112:
URL: https://github.com/apache/airflow/pull/77112#discussion_r4582050588
##########
providers/google/src/airflow/providers/google/cloud/triggers/cloud_batch.py:
##########
@@ -20,13 +20,34 @@
from collections.abc import AsyncIterator, Sequence
from typing import Any
+from google.api_core.exceptions import (
+ Aborted,
+ DeadlineExceeded,
+ GatewayTimeout,
+ InternalServerError,
+ ResourceExhausted,
+ ServiceUnavailable,
+)
from google.cloud.batch_v1 import Job, JobStatus
from airflow.providers.google.cloud.hooks.cloud_batch import
CloudBatchAsyncHook
from airflow.triggers.base import BaseTrigger, TriggerEvent
DEFAULT_BATCH_LOCATION = "us-central1"
+# gRPC errors that indicate a transient failure of the Cloud Batch API rather
than a terminal
+# state of the job. The default retry of ``BatchServiceAsyncClient.get_job``
only covers
+# ServiceUnavailable, so the others would otherwise fail the deferred task
while the job keeps
+# running, and the task-level retry would submit the job again.
+_RETRYABLE_GRPC_EXCEPTIONS = (
+ ServiceUnavailable, # 503 / UNAVAILABLE
+ InternalServerError, # 500 / INTERNAL
+ DeadlineExceeded, # 504 / DEADLINE_EXCEEDED
+ GatewayTimeout, # 504 / gateway timeout
+ ResourceExhausted, # 429 / RESOURCE_EXHAUSTED
+ Aborted, # ABORTED
+)
Review Comment:
the SDK's default retry on get_job already retries ServiceUnavailable for
60s and then raises RetryError (not ServiceUnavailable), so a 503 outage longer
than that still skips this branch and fails the task. catching RetryError too
covers it (needs `RetryError` added to the `google.api_core.exceptions` import
above):
```suggestion
Aborted, # ABORTED
# get_job's default retry wraps a ServiceUnavailable that outlasts its
60s deadline in RetryError.
RetryError,
)
```
fixed this plus a test in this commit, you can just cherry pick it instead
of redoing it:
https://github.com/its-amann/airflow/commit/85249ded25902a1be04e0bc1db8beee5e3201110
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]