darrenleeweber commented on a change in pull request #5825: [AIRFLOW-5218] less polling for AWS Batch status URL: https://github.com/apache/airflow/pull/5825#discussion_r314853819
########## File path: airflow/contrib/operators/awsbatch_operator.py ########## @@ -133,16 +134,25 @@ def _wait_for_task_ended(self): retry = True retries = 0 - while retries < self.max_retries and retry: - self.log.info('AWS Batch retry in the next %s seconds', retries) - response = self.client.describe_jobs( - jobs=[self.jobId] - ) - if response['jobs'][-1]['status'] in ['SUCCEEDED', 'FAILED']: + # Allow a batch job a minute to spin up. A random interval + # decreases the chances of exceeding an AWS API throttle + # limit when there are many concurrent tasks. + sleep(randint(10, 60)) Review comment: Modified the range to (5, 30) sec in amended commit ---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services