[
https://issues.apache.org/jira/browse/AIRFLOW-6245?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17010571#comment-17010571
]
ASF subversion and git services commented on AIRFLOW-6245:
----------------------------------------------------------
Commit 78d8fe6944b689b9b0af99255286e34e06eedec3 in airflow's branch
refs/heads/master from Darren Weber
[ https://gitbox.apache.org/repos/asf?p=airflow.git;h=78d8fe6 ]
[AIRFLOW-6245] Add custom waiters for AWS batch jobs (#6811)
- add AwsBatchWaiters
- the waiters are based on botocore, but not yet
available for AWS batch services
- refactor AwsBatchOperator:
- use an optional waiters object to wait for
batch job status indicators
- split execute into submit_job and monitor_job
- use job_id with an optional init-parameter;
discard jobId and jobName (already has job_name)
- inherit from AwsBatchClient
- add notes to UPDATING.md
- extract class for AwsBatchClient
- move responsibility for batch API calls and
response parsing to this client
- move responsibility for default wait and
polling to this client
- rename BatchProtocol to AwsBatchProtocol [AIP-21]
- test backward compatibility
- add PROTOCOLS to tests/test_core_to_contrib.py
- add notes to UPDATING.md
- split up polling for job status into steps:
- poll for a JobExists
- poll for a JobRunning
- poll for a JobComplete
- use random jitter for wait-polling delays for
high concurrency job polling
- modify the exponential backoff delay for the
existing polling functions
- revise and update unit tests
> Allow custom waiters for AWS batch jobs
> ---------------------------------------
>
> Key: AIRFLOW-6245
> URL: https://issues.apache.org/jira/browse/AIRFLOW-6245
> Project: Apache Airflow
> Issue Type: Improvement
> Components: aws
> Affects Versions: 1.10.6
> Reporter: Darren Weber
> Assignee: Darren Weber
> Priority: Minor
> Labels: AWS, aws-batch
> Fix For: 2.0.0, 1.10.8
>
>
> The botocore waiter for AWS batch jobs has not been merged and released for
> several years, i.e.
> - [https://github.com/boto/botocore/pull/1307]
> While working on this Airflow issue, I've also pushed up a PR on botocore to
> use a default waiter with exponential backoff and add an option to use a
> custom function for the delays between polling status, see
> - [https://github.com/boto/botocore/issues/1915]
> - [https://github.com/boto/botocore/pull/1921]
>
> For Airflow, adopt something from botocore PR-1307 as an example to create a
> default batch job waiter in Airflow. As a guide to creating a custom waiter,
> see
> - [https://www.2ndwatch.com/blog/use-waiters-boto3-write/]
> -
> [https://boto3.amazonaws.com/v1/documentation/api/latest/guide/clients.html#waiters]
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
