[
https://issues.apache.org/jira/browse/AIRFLOW-3156?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16642949#comment-16642949
]
Jacopo Sabbatini edited comment on AIRFLOW-3156 at 10/9/18 8:24 AM:
--------------------------------------------------------------------
Well in a sense it's like the use case for the Mesos and Kubernetes executor.
Correct me if wrong but when using the AWS Batch operator all the task does is
continuously polling for the task status until done. This seems a very
inefficient use of a worker. Furthermore if I wanted to submit many tasks using
the Batch operator I suspect I would start seeing the polling API requests
being throttled by AWS.
Having a Batch executor it would mean that the polling of the job status can be
batched using the DescribeJobs Batch API this would make having many tasks more
efficient. Using Batch as executor would also simplify deployments since it
remove the needs for workers that are now managed by AWS Batch.
was (Author: dedalusj):
Well in a sense it's like the use case for the Mesos and Kubernetes executor.
Correct me if wrong but when using the AWS Batch operator all the task does is
continuously polling for the task status until done. This seems a very
inefficient use of a worker. Furthermore if I wanted to submit many tasks using
the Batch operator I suspect I would start seeing the polling API requests
being throttled by AWS.
Having a Batch executor it would mean that the polling of the job status can be
batched using the DescribeJobs Batch API this would make having many tasks more
efficient.
> AWS Batch executor
> ------------------
>
> Key: AIRFLOW-3156
> URL: https://issues.apache.org/jira/browse/AIRFLOW-3156
> Project: Apache Airflow
> Issue Type: New Feature
> Components: executor
> Reporter: Jacopo Sabbatini
> Priority: Major
> Labels: aws
>
> Add support for AWS Batch as executor.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
