shahar1 commented on code in PR #40863:
URL: https://github.com/apache/airflow/pull/40863#discussion_r1855178999
##########
airflow/providers/google/cloud/operators/bigquery.py:
##########
@@ -3021,7 +3021,15 @@ def execute(self, context: Any):
# Wait for the job to complete
if not self.deferrable:
job.result(timeout=self.result_timeout, retry=self.result_retry)
- self._handle_job_error(job)
+ while True:
+ if job.state in ("PENDING", "RUNNING"):
+ import time
+
+ time.sleep(5)
Review Comment:
@RNHTTR @kandharvishnu In a second thought, maybe it's better to have a
`time.sleep()` - otherwise, if it is pending or running for long time -
requests will be sent immedately one after another, which might cause exceeding
API rate limit.
Example from another Google providers:
https://github.com/apache/airflow/blob/3c58e01266f884544fdebc70f92b63848c610d2d/providers/src/airflow/providers/google/cloud/hooks/dataproc.py#L1241
What do you guys think?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
