darrenleeweber commented on a change in pull request #5825: [AIRFLOW-5218] less
polling for AWS Batch status
URL: https://github.com/apache/airflow/pull/5825#discussion_r314846637
##########
File path: airflow/contrib/operators/awsbatch_operator.py
##########
@@ -133,16 +134,25 @@ def _wait_for_task_ended(self):
retry = True
retries = 0
- while retries < self.max_retries and retry:
- self.log.info('AWS Batch retry in the next %s seconds',
retries)
- response = self.client.describe_jobs(
- jobs=[self.jobId]
- )
- if response['jobs'][-1]['status'] in ['SUCCEEDED', 'FAILED']:
+ # Allow a batch job a minute to spin up. A random interval
+ # decreases the chances of exceeding an AWS API throttle
+ # limit when there are many concurrent tasks.
+ sleep(randint(10, 60))
+
+ while True:
+ response = self.client.describe_jobs(jobs=[self.jobId])
+ status = response['jobs'][-1]['status']
+ self.log.info('AWS Batch status: %s', status)
+ if status in ['SUCCEEDED', 'FAILED']:
retry = False
Review comment:
Thanks for the loop review and suggestion, will see if some of that can be
adopted in an amended commit - but note that the suggestion now removes the
status update to the `retry` variable so that bit needs a tweak.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
