abhishekshenoy edited a comment on pull request #19740:
URL: https://github.com/apache/airflow/pull/19740#issuecomment-989716007
@lwyszomi @potiuk This will solve the server error received at the start of
the job. We are experiencing issues wherein Google apis fail sometimes with
connectivity issues .
On connecting with Google Support they mentioned :
`Users are expected to see this set of errors (503) every now and then.
These could be due to expected issues like busy servers, network
unavailability, etc`
In the above scenarios as the response is not successfully retrieved , Job
Sensor fails though the cluster is running.
I have an approach which is similar to the one used here but by using
something like a resettable counter for any 'API Reponse Error' which on
passing a threshold , only then should fail the Job Sensor.
```
File
"/home/airflow/.local/lib/python3.8/site-packages/airflow/providers/google/cloud/sensors/dataproc.py",
line 63, in poke
job = hook.get_job(job_id=self.dataproc_job_id, location=self.location,
project_id=self.project_id)
.
.
.
google.api_core.exceptions.ServiceUnavailable: 503 The service is currently
unavailable.
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
