darrenleeweber commented on a change in pull request #5825: [AIRFLOW-5218] less
polling for AWS Batch status
URL: https://github.com/apache/airflow/pull/5825#discussion_r314845135
##########
File path: airflow/contrib/operators/awsbatch_operator.py
##########
@@ -133,16 +134,25 @@ def _wait_for_task_ended(self):
retry = True
retries = 0
- while retries < self.max_retries and retry:
- self.log.info('AWS Batch retry in the next %s seconds',
retries)
- response = self.client.describe_jobs(
- jobs=[self.jobId]
- )
- if response['jobs'][-1]['status'] in ['SUCCEEDED', 'FAILED']:
+ # Allow a batch job a minute to spin up. A random interval
+ # decreases the chances of exceeding an AWS API throttle
+ # limit when there are many concurrent tasks.
+ sleep(randint(10, 60))
Review comment:
Good question, points to note:
- the job is already submitted, so this does not delay _this_ job, only a
delay in checking it's status (assuming it takes a while to kick it off after
the first API request to start the job)
- this assumes the BatchOperator only applies to batch jobs (not Glue) and
that each batch job is not expected to provide anything like near-real-time
operations where the next step in the DAG depends on checking this batch job
status
- there is a change to the loop logic that avoids an extra sleep when it's
not required (but that's not exactly relevant to this concern)
- certainly there is a somewhat arbitrary range applied here, at present,
and opinions on the (min, max) range for this are welcome (like dropping the
min?)
- the JIRA issue notes that better solutions might be possible, like using a
reschedule for the task rather than polling, but that's not the focus on this PR
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
