[jira] [Commented] (AIRFLOW-6440) AWS Fargate Executor (AIP-29) (WIP)
[
https://issues.apache.org/jira/browse/AIRFLOW-6440?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17241758#comment-17241758
]
ASF GitHub Bot commented on AIRFLOW-6440:
-
potiuk commented on pull request #7030:
URL: https://github.com/apache/airflow/pull/7030#issuecomment-736732600
> For the time-being this is an optional plugin outside of the Airflow Core
repo. Here you'll find AWS Batch, AWS ECS, and AWS Fargate Executors. I
strongly recommend AWS Batch!
>
> **pip install airflow-aws-executors**
Nice!
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
> AWS Fargate Executor (AIP-29) (WIP)
> ---
>
> Key: AIRFLOW-6440
> URL: https://issues.apache.org/jira/browse/AIRFLOW-6440
> Project: Apache Airflow
> Issue Type: Improvement
> Components: aws, executors
>Affects Versions: 1.10.8
> Environment: AWS Cloud
>Reporter: Ahmed Elzeiny
>Assignee: Ahmed Elzeiny
>Priority: Minor
> Labels: AWS, Executor, autoscaling
> Original Estimate: 336h
> Remaining Estimate: 336h
>
> h1. Links
> AIP -
> [https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-29%3A+AWS+Fargate+Executor]
> PR - [https://github.com/apache/airflow/pull/7030]
> h1. Airflow on AWS Fargate
> {color:#707070}We propose the creation of a new Airflow Executor, called the
> FargateExecutor, that runs tasks asynchronously on AWS Fargate. The Airflow
> Scheduler comes up with a command that needs to be executed in some shell. A
> Docker container parameterized with the command is passed in as an ARG, and
> AWS Fargate provisions a new instance with . The container then completes or
> fails the job, causing the container to die along with the Fargate instance.
> The executor is responsible for keeping track what happened to the task with
> an airflow task id and AWS ARN number, and based off of the instance exit
> code we either say that the task succeeded or failed.{color}
> h1. Proposed Implementation
> As you could probably deduce, the underlying mechanism to launch, track, and
> stop Fargate instances is AWS' Boto3 Library.
> To accomplish this we create a FargateExecutor under the "airflow.executors"
> module. This class will extend from BaseExecutor and override 5 methods:
> {{start()}}, {{{color:#3366ff}sync(){color}}},{{{color:#3366ff}
> execute_async(){color}}}, {{{color:#3366ff}end(){color}}}, and
> {{{color:#3366ff}terminate(){color}}}. Internally, the FargateExecutor uses
> boto3 for monitoring and deployment purposes.
> {color:#707070}The three major Boto3 API calls are:{color}
> * {color:#707070}The {color:#0747a6}{{execute_async()}}{color} function
> calls boto3's {{{color:#0747a6}run_task(){color}}} function.{color}
> * {color:#707070}The {{{color:#0747a6}sync(){color}}} function calls boto3's
> {{{color:#0747a6}describe_tasks(){color}}} function.{color}
> * {color:#707070}The {{{color:#0747a6}terminate(){color}}} function calls
> boto3's {{{color:#0747a6}stop_task(){color}}} function.{color}
> h1. Maintenance
> The executor itself is nothing special since it mostly relies on overriding
> the proper methods from .
> In general, AWS is fairly committed to keeping their APIs in service. Fargate
> is rather new and I've personally perceived a lot more features added as
> optional parameters over the course of the past year. However, the required
> parameters for the three Boto3 calls that are used have remained the same.
> I've also written test-cases that ensures that the Boto3 calls made are
> complaint to the most current version of their APIs.
> We've also introduced a callback hook (very similar to the Celery Executor)
> that allows users to launch tasks with their own parameters. Therefore if a
> user doesn't like the default parameter options used in Boto3's
> \{{run_task(),}}then they can call it themselves with whatever parameters
> they want. This means that Airflow doesn't have to add a new configuration
> everytime AWS makes an addition to AWS Fargate. It's just one configuration
> to cover them all.
> h1. {color:#707070}Proposed Configuration{color}
>
> {code:java}
> [fargate]
> # For more information on any of these execution parameters, see the link
> below:
> #
> https://boto3.amazonaws.com/v1/documentation/api/latest/reference/services/ecs.html#ECS.Client.run_task
> # For boto3 credential management, see
> #
> https://boto3.amazonaws.com/v1/documentation/api/latest/guide/configuration.html
> ### MANDATORY CONFIGS:
> # Name of region
> region = us-west-2
> # Name of
[jira] [Commented] (AIRFLOW-6440) AWS Fargate Executor (AIP-29) (WIP)
[
https://issues.apache.org/jira/browse/AIRFLOW-6440?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17241746#comment-17241746
]
ASF GitHub Bot commented on AIRFLOW-6440:
-
aelzeiny edited a comment on pull request #7030:
URL: https://github.com/apache/airflow/pull/7030#issuecomment-736721648
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
> AWS Fargate Executor (AIP-29) (WIP)
> ---
>
> Key: AIRFLOW-6440
> URL: https://issues.apache.org/jira/browse/AIRFLOW-6440
> Project: Apache Airflow
> Issue Type: Improvement
> Components: aws, executors
>Affects Versions: 1.10.8
> Environment: AWS Cloud
>Reporter: Ahmed Elzeiny
>Assignee: Ahmed Elzeiny
>Priority: Minor
> Labels: AWS, Executor, autoscaling
> Original Estimate: 336h
> Remaining Estimate: 336h
>
> h1. Links
> AIP -
> [https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-29%3A+AWS+Fargate+Executor]
> PR - [https://github.com/apache/airflow/pull/7030]
> h1. Airflow on AWS Fargate
> {color:#707070}We propose the creation of a new Airflow Executor, called the
> FargateExecutor, that runs tasks asynchronously on AWS Fargate. The Airflow
> Scheduler comes up with a command that needs to be executed in some shell. A
> Docker container parameterized with the command is passed in as an ARG, and
> AWS Fargate provisions a new instance with . The container then completes or
> fails the job, causing the container to die along with the Fargate instance.
> The executor is responsible for keeping track what happened to the task with
> an airflow task id and AWS ARN number, and based off of the instance exit
> code we either say that the task succeeded or failed.{color}
> h1. Proposed Implementation
> As you could probably deduce, the underlying mechanism to launch, track, and
> stop Fargate instances is AWS' Boto3 Library.
> To accomplish this we create a FargateExecutor under the "airflow.executors"
> module. This class will extend from BaseExecutor and override 5 methods:
> {{start()}}, {{{color:#3366ff}sync(){color}}},{{{color:#3366ff}
> execute_async(){color}}}, {{{color:#3366ff}end(){color}}}, and
> {{{color:#3366ff}terminate(){color}}}. Internally, the FargateExecutor uses
> boto3 for monitoring and deployment purposes.
> {color:#707070}The three major Boto3 API calls are:{color}
> * {color:#707070}The {color:#0747a6}{{execute_async()}}{color} function
> calls boto3's {{{color:#0747a6}run_task(){color}}} function.{color}
> * {color:#707070}The {{{color:#0747a6}sync(){color}}} function calls boto3's
> {{{color:#0747a6}describe_tasks(){color}}} function.{color}
> * {color:#707070}The {{{color:#0747a6}terminate(){color}}} function calls
> boto3's {{{color:#0747a6}stop_task(){color}}} function.{color}
> h1. Maintenance
> The executor itself is nothing special since it mostly relies on overriding
> the proper methods from .
> In general, AWS is fairly committed to keeping their APIs in service. Fargate
> is rather new and I've personally perceived a lot more features added as
> optional parameters over the course of the past year. However, the required
> parameters for the three Boto3 calls that are used have remained the same.
> I've also written test-cases that ensures that the Boto3 calls made are
> complaint to the most current version of their APIs.
> We've also introduced a callback hook (very similar to the Celery Executor)
> that allows users to launch tasks with their own parameters. Therefore if a
> user doesn't like the default parameter options used in Boto3's
> \{{run_task(),}}then they can call it themselves with whatever parameters
> they want. This means that Airflow doesn't have to add a new configuration
> everytime AWS makes an addition to AWS Fargate. It's just one configuration
> to cover them all.
> h1. {color:#707070}Proposed Configuration{color}
>
> {code:java}
> [fargate]
> # For more information on any of these execution parameters, see the link
> below:
> #
> https://boto3.amazonaws.com/v1/documentation/api/latest/reference/services/ecs.html#ECS.Client.run_task
> # For boto3 credential management, see
> #
> https://boto3.amazonaws.com/v1/documentation/api/latest/guide/configuration.html
> ### MANDATORY CONFIGS:
> # Name of region
> region = us-west-2
> # Name of cluster
> cluster = test-airflow
> ### EITHER POPULATE THESE:
> # Name of task definition with a bootable-container. Note that this container
> will receive an airflow CLI
> # command as an additional parameter to its entrypoint. It's job is to
[jira] [Commented] (AIRFLOW-6440) AWS Fargate Executor (AIP-29) (WIP)
[
https://issues.apache.org/jira/browse/AIRFLOW-6440?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17241744#comment-17241744
]
ASF GitHub Bot commented on AIRFLOW-6440:
-
aelzeiny commented on pull request #7030:
URL: https://github.com/apache/airflow/pull/7030#issuecomment-736721648
The AWS Batch, AWS ECS, and AWS Fargate Executors[ have been moved to this
Github.]( https://github.com/aelzeiny/airflow-aws-executors)
pip install airflow-aws-executors
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
> AWS Fargate Executor (AIP-29) (WIP)
> ---
>
> Key: AIRFLOW-6440
> URL: https://issues.apache.org/jira/browse/AIRFLOW-6440
> Project: Apache Airflow
> Issue Type: Improvement
> Components: aws, executors
>Affects Versions: 1.10.8
> Environment: AWS Cloud
>Reporter: Ahmed Elzeiny
>Assignee: Ahmed Elzeiny
>Priority: Minor
> Labels: AWS, Executor, autoscaling
> Original Estimate: 336h
> Remaining Estimate: 336h
>
> h1. Links
> AIP -
> [https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-29%3A+AWS+Fargate+Executor]
> PR - [https://github.com/apache/airflow/pull/7030]
> h1. Airflow on AWS Fargate
> {color:#707070}We propose the creation of a new Airflow Executor, called the
> FargateExecutor, that runs tasks asynchronously on AWS Fargate. The Airflow
> Scheduler comes up with a command that needs to be executed in some shell. A
> Docker container parameterized with the command is passed in as an ARG, and
> AWS Fargate provisions a new instance with . The container then completes or
> fails the job, causing the container to die along with the Fargate instance.
> The executor is responsible for keeping track what happened to the task with
> an airflow task id and AWS ARN number, and based off of the instance exit
> code we either say that the task succeeded or failed.{color}
> h1. Proposed Implementation
> As you could probably deduce, the underlying mechanism to launch, track, and
> stop Fargate instances is AWS' Boto3 Library.
> To accomplish this we create a FargateExecutor under the "airflow.executors"
> module. This class will extend from BaseExecutor and override 5 methods:
> {{start()}}, {{{color:#3366ff}sync(){color}}},{{{color:#3366ff}
> execute_async(){color}}}, {{{color:#3366ff}end(){color}}}, and
> {{{color:#3366ff}terminate(){color}}}. Internally, the FargateExecutor uses
> boto3 for monitoring and deployment purposes.
> {color:#707070}The three major Boto3 API calls are:{color}
> * {color:#707070}The {color:#0747a6}{{execute_async()}}{color} function
> calls boto3's {{{color:#0747a6}run_task(){color}}} function.{color}
> * {color:#707070}The {{{color:#0747a6}sync(){color}}} function calls boto3's
> {{{color:#0747a6}describe_tasks(){color}}} function.{color}
> * {color:#707070}The {{{color:#0747a6}terminate(){color}}} function calls
> boto3's {{{color:#0747a6}stop_task(){color}}} function.{color}
> h1. Maintenance
> The executor itself is nothing special since it mostly relies on overriding
> the proper methods from .
> In general, AWS is fairly committed to keeping their APIs in service. Fargate
> is rather new and I've personally perceived a lot more features added as
> optional parameters over the course of the past year. However, the required
> parameters for the three Boto3 calls that are used have remained the same.
> I've also written test-cases that ensures that the Boto3 calls made are
> complaint to the most current version of their APIs.
> We've also introduced a callback hook (very similar to the Celery Executor)
> that allows users to launch tasks with their own parameters. Therefore if a
> user doesn't like the default parameter options used in Boto3's
> \{{run_task(),}}then they can call it themselves with whatever parameters
> they want. This means that Airflow doesn't have to add a new configuration
> everytime AWS makes an addition to AWS Fargate. It's just one configuration
> to cover them all.
> h1. {color:#707070}Proposed Configuration{color}
>
> {code:java}
> [fargate]
> # For more information on any of these execution parameters, see the link
> below:
> #
> https://boto3.amazonaws.com/v1/documentation/api/latest/reference/services/ecs.html#ECS.Client.run_task
> # For boto3 credential management, see
> #
> https://boto3.amazonaws.com/v1/documentation/api/latest/guide/configuration.html
> ### MANDATORY CONFIGS:
> # Name of region
> region = us-west-2
> # Name of cluster
> cluster = test-airflow
> ### EITHER POPULATE THESE:
> # Name
[jira] [Commented] (AIRFLOW-6440) AWS Fargate Executor (AIP-29) (WIP)
[
https://issues.apache.org/jira/browse/AIRFLOW-6440?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17173596#comment-17173596
]
ASF GitHub Bot commented on AIRFLOW-6440:
-
stale[bot] closed pull request #7030:
URL: https://github.com/apache/airflow/pull/7030
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
> AWS Fargate Executor (AIP-29) (WIP)
> ---
>
> Key: AIRFLOW-6440
> URL: https://issues.apache.org/jira/browse/AIRFLOW-6440
> Project: Apache Airflow
> Issue Type: Improvement
> Components: aws, executors
>Affects Versions: 1.10.8
> Environment: AWS Cloud
>Reporter: Ahmed Elzeiny
>Assignee: Ahmed Elzeiny
>Priority: Minor
> Labels: AWS, Executor, autoscaling
> Original Estimate: 336h
> Remaining Estimate: 336h
>
> h1. Links
> AIP -
> [https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-29%3A+AWS+Fargate+Executor]
> PR - [https://github.com/apache/airflow/pull/7030]
> h1. Airflow on AWS Fargate
> {color:#707070}We propose the creation of a new Airflow Executor, called the
> FargateExecutor, that runs tasks asynchronously on AWS Fargate. The Airflow
> Scheduler comes up with a command that needs to be executed in some shell. A
> Docker container parameterized with the command is passed in as an ARG, and
> AWS Fargate provisions a new instance with . The container then completes or
> fails the job, causing the container to die along with the Fargate instance.
> The executor is responsible for keeping track what happened to the task with
> an airflow task id and AWS ARN number, and based off of the instance exit
> code we either say that the task succeeded or failed.{color}
> h1. Proposed Implementation
> As you could probably deduce, the underlying mechanism to launch, track, and
> stop Fargate instances is AWS' Boto3 Library.
> To accomplish this we create a FargateExecutor under the "airflow.executors"
> module. This class will extend from BaseExecutor and override 5 methods:
> {{start()}}, {{{color:#3366ff}sync(){color}}},{{{color:#3366ff}
> execute_async(){color}}}, {{{color:#3366ff}end(){color}}}, and
> {{{color:#3366ff}terminate(){color}}}. Internally, the FargateExecutor uses
> boto3 for monitoring and deployment purposes.
> {color:#707070}The three major Boto3 API calls are:{color}
> * {color:#707070}The {color:#0747a6}{{execute_async()}}{color} function
> calls boto3's {{{color:#0747a6}run_task(){color}}} function.{color}
> * {color:#707070}The {{{color:#0747a6}sync(){color}}} function calls boto3's
> {{{color:#0747a6}describe_tasks(){color}}} function.{color}
> * {color:#707070}The {{{color:#0747a6}terminate(){color}}} function calls
> boto3's {{{color:#0747a6}stop_task(){color}}} function.{color}
> h1. Maintenance
> The executor itself is nothing special since it mostly relies on overriding
> the proper methods from .
> In general, AWS is fairly committed to keeping their APIs in service. Fargate
> is rather new and I've personally perceived a lot more features added as
> optional parameters over the course of the past year. However, the required
> parameters for the three Boto3 calls that are used have remained the same.
> I've also written test-cases that ensures that the Boto3 calls made are
> complaint to the most current version of their APIs.
> We've also introduced a callback hook (very similar to the Celery Executor)
> that allows users to launch tasks with their own parameters. Therefore if a
> user doesn't like the default parameter options used in Boto3's
> \{{run_task(),}}then they can call it themselves with whatever parameters
> they want. This means that Airflow doesn't have to add a new configuration
> everytime AWS makes an addition to AWS Fargate. It's just one configuration
> to cover them all.
> h1. {color:#707070}Proposed Configuration{color}
>
> {code:java}
> [fargate]
> # For more information on any of these execution parameters, see the link
> below:
> #
> https://boto3.amazonaws.com/v1/documentation/api/latest/reference/services/ecs.html#ECS.Client.run_task
> # For boto3 credential management, see
> #
> https://boto3.amazonaws.com/v1/documentation/api/latest/guide/configuration.html
> ### MANDATORY CONFIGS:
> # Name of region
> region = us-west-2
> # Name of cluster
> cluster = test-airflow
> ### EITHER POPULATE THESE:
> # Name of task definition with a bootable-container. Note that this container
> will receive an airflow CLI
> # command as an additional parameter to its entrypoint. It's job is to
> boot-up and run this command
[jira] [Commented] (AIRFLOW-6440) AWS Fargate Executor (AIP-29) (WIP)
[
https://issues.apache.org/jira/browse/AIRFLOW-6440?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17166994#comment-17166994
]
ASF GitHub Bot commented on AIRFLOW-6440:
-
stale[bot] commented on pull request #7030:
URL: https://github.com/apache/airflow/pull/7030#issuecomment-665446972
This issue has been automatically marked as stale because it has not had
recent activity. It will be closed if no further activity occurs. Thank you for
your contributions.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
> AWS Fargate Executor (AIP-29) (WIP)
> ---
>
> Key: AIRFLOW-6440
> URL: https://issues.apache.org/jira/browse/AIRFLOW-6440
> Project: Apache Airflow
> Issue Type: Improvement
> Components: aws, executors
>Affects Versions: 1.10.8
> Environment: AWS Cloud
>Reporter: Ahmed Elzeiny
>Assignee: Ahmed Elzeiny
>Priority: Minor
> Labels: AWS, Executor, autoscaling
> Original Estimate: 336h
> Remaining Estimate: 336h
>
> h1. Links
> AIP -
> [https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-29%3A+AWS+Fargate+Executor]
> PR - [https://github.com/apache/airflow/pull/7030]
> h1. Airflow on AWS Fargate
> {color:#707070}We propose the creation of a new Airflow Executor, called the
> FargateExecutor, that runs tasks asynchronously on AWS Fargate. The Airflow
> Scheduler comes up with a command that needs to be executed in some shell. A
> Docker container parameterized with the command is passed in as an ARG, and
> AWS Fargate provisions a new instance with . The container then completes or
> fails the job, causing the container to die along with the Fargate instance.
> The executor is responsible for keeping track what happened to the task with
> an airflow task id and AWS ARN number, and based off of the instance exit
> code we either say that the task succeeded or failed.{color}
> h1. Proposed Implementation
> As you could probably deduce, the underlying mechanism to launch, track, and
> stop Fargate instances is AWS' Boto3 Library.
> To accomplish this we create a FargateExecutor under the "airflow.executors"
> module. This class will extend from BaseExecutor and override 5 methods:
> {{start()}}, {{{color:#3366ff}sync(){color}}},{{{color:#3366ff}
> execute_async(){color}}}, {{{color:#3366ff}end(){color}}}, and
> {{{color:#3366ff}terminate(){color}}}. Internally, the FargateExecutor uses
> boto3 for monitoring and deployment purposes.
> {color:#707070}The three major Boto3 API calls are:{color}
> * {color:#707070}The {color:#0747a6}{{execute_async()}}{color} function
> calls boto3's {{{color:#0747a6}run_task(){color}}} function.{color}
> * {color:#707070}The {{{color:#0747a6}sync(){color}}} function calls boto3's
> {{{color:#0747a6}describe_tasks(){color}}} function.{color}
> * {color:#707070}The {{{color:#0747a6}terminate(){color}}} function calls
> boto3's {{{color:#0747a6}stop_task(){color}}} function.{color}
> h1. Maintenance
> The executor itself is nothing special since it mostly relies on overriding
> the proper methods from .
> In general, AWS is fairly committed to keeping their APIs in service. Fargate
> is rather new and I've personally perceived a lot more features added as
> optional parameters over the course of the past year. However, the required
> parameters for the three Boto3 calls that are used have remained the same.
> I've also written test-cases that ensures that the Boto3 calls made are
> complaint to the most current version of their APIs.
> We've also introduced a callback hook (very similar to the Celery Executor)
> that allows users to launch tasks with their own parameters. Therefore if a
> user doesn't like the default parameter options used in Boto3's
> \{{run_task(),}}then they can call it themselves with whatever parameters
> they want. This means that Airflow doesn't have to add a new configuration
> everytime AWS makes an addition to AWS Fargate. It's just one configuration
> to cover them all.
> h1. {color:#707070}Proposed Configuration{color}
>
> {code:java}
> [fargate]
> # For more information on any of these execution parameters, see the link
> below:
> #
> https://boto3.amazonaws.com/v1/documentation/api/latest/reference/services/ecs.html#ECS.Client.run_task
> # For boto3 credential management, see
> #
> https://boto3.amazonaws.com/v1/documentation/api/latest/guide/configuration.html
> ### MANDATORY CONFIGS:
> # Name of region
> region = us-west-2
> # Name of cluster
> cluster = test-airflow
> ### EITHER POPULATE THESE:
> # Name
[jira] [Commented] (AIRFLOW-6440) AWS Fargate Executor (AIP-29) (WIP)
[ https://issues.apache.org/jira/browse/AIRFLOW-6440?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17134748#comment-17134748 ] ASF GitHub Bot commented on AIRFLOW-6440: - codecov-commenter commented on pull request #7030: URL: https://github.com/apache/airflow/pull/7030#issuecomment-643607992 # [Codecov](https://codecov.io/gh/apache/airflow/pull/7030?src=pr=h1) Report > Merging [#7030](https://codecov.io/gh/apache/airflow/pull/7030?src=pr=desc) into [master](https://codecov.io/gh/apache/airflow/commit/749fc473e490bdec859469cd604e038f531f6601=desc) will **decrease** coverage by `0.24%`. > The diff coverage is `92.82%`. [](https://codecov.io/gh/apache/airflow/pull/7030?src=pr=tree) ```diff @@Coverage Diff @@ ## master#7030 +/- ## == - Coverage 86.91% 86.66% -0.25% == Files 920 922 +2 Lines 4433644611 +275 == + Hits3853338662 +129 - Misses 5803 5949 +146 ``` | [Impacted Files](https://codecov.io/gh/apache/airflow/pull/7030?src=pr=tree) | Coverage Δ | | |---|---|---| | [airflow/config\_templates/default\_aws\_ecs.py](https://codecov.io/gh/apache/airflow/pull/7030/diff?src=pr=tree#diff-YWlyZmxvdy9jb25maWdfdGVtcGxhdGVzL2RlZmF1bHRfYXdzX2Vjcy5weQ==) | `85.71% <85.71%> (ø)` | | | [airflow/executors/aws\_ecs\_executor.py](https://codecov.io/gh/apache/airflow/pull/7030/diff?src=pr=tree#diff-YWlyZmxvdy9leGVjdXRvcnMvYXdzX2Vjc19leGVjdXRvci5weQ==) | `93.02% <93.02%> (ø)` | | | [airflow/executors/executor\_loader.py](https://codecov.io/gh/apache/airflow/pull/7030/diff?src=pr=tree#diff-YWlyZmxvdy9leGVjdXRvcnMvZXhlY3V0b3JfbG9hZGVyLnB5) | `100.00% <100.00%> (ø)` | | | [airflow/kubernetes/volume\_mount.py](https://codecov.io/gh/apache/airflow/pull/7030/diff?src=pr=tree#diff-YWlyZmxvdy9rdWJlcm5ldGVzL3ZvbHVtZV9tb3VudC5weQ==) | `44.44% <0.00%> (-55.56%)` | :arrow_down: | | [airflow/kubernetes/volume.py](https://codecov.io/gh/apache/airflow/pull/7030/diff?src=pr=tree#diff-YWlyZmxvdy9rdWJlcm5ldGVzL3ZvbHVtZS5weQ==) | `52.94% <0.00%> (-47.06%)` | :arrow_down: | | [airflow/kubernetes/pod\_launcher.py](https://codecov.io/gh/apache/airflow/pull/7030/diff?src=pr=tree#diff-YWlyZmxvdy9rdWJlcm5ldGVzL3BvZF9sYXVuY2hlci5weQ==) | `47.18% <0.00%> (-45.08%)` | :arrow_down: | | [...viders/cncf/kubernetes/operators/kubernetes\_pod.py](https://codecov.io/gh/apache/airflow/pull/7030/diff?src=pr=tree#diff-YWlyZmxvdy9wcm92aWRlcnMvY25jZi9rdWJlcm5ldGVzL29wZXJhdG9ycy9rdWJlcm5ldGVzX3BvZC5weQ==) | `69.69% <0.00%> (-25.26%)` | :arrow_down: | | [airflow/kubernetes/refresh\_config.py](https://codecov.io/gh/apache/airflow/pull/7030/diff?src=pr=tree#diff-YWlyZmxvdy9rdWJlcm5ldGVzL3JlZnJlc2hfY29uZmlnLnB5) | `50.98% <0.00%> (-23.53%)` | :arrow_down: | | [airflow/plugins\_manager.py](https://codecov.io/gh/apache/airflow/pull/7030/diff?src=pr=tree#diff-YWlyZmxvdy9wbHVnaW5zX21hbmFnZXIucHk=) | `88.58% <0.00%> (-1.56%)` | :arrow_down: | | [airflow/serialization/serialized\_objects.py](https://codecov.io/gh/apache/airflow/pull/7030/diff?src=pr=tree#diff-YWlyZmxvdy9zZXJpYWxpemF0aW9uL3NlcmlhbGl6ZWRfb2JqZWN0cy5weQ==) | `90.09% <0.00%> (-0.52%)` | :arrow_down: | | ... and [3 more](https://codecov.io/gh/apache/airflow/pull/7030/diff?src=pr=tree-more) | | -- [Continue to review full report at Codecov](https://codecov.io/gh/apache/airflow/pull/7030?src=pr=continue). > **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta) > `Δ = absolute (impact)`, `ø = not affected`, `? = missing data` > Powered by [Codecov](https://codecov.io/gh/apache/airflow/pull/7030?src=pr=footer). Last update [749fc47...c9a2d19](https://codecov.io/gh/apache/airflow/pull/7030?src=pr=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org > AWS Fargate Executor (AIP-29) (WIP) > --- > > Key: AIRFLOW-6440 > URL: https://issues.apache.org/jira/browse/AIRFLOW-6440 > Project: Apache Airflow > Issue Type: Improvement > Components: aws, executors >Affects Versions: 1.10.8 > Environment: AWS Cloud >
[jira] [Commented] (AIRFLOW-6440) AWS Fargate Executor (AIP-29) (WIP)
[
https://issues.apache.org/jira/browse/AIRFLOW-6440?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17098795#comment-17098795
]
ASF GitHub Bot commented on AIRFLOW-6440:
-
feluelle edited a comment on pull request #7030:
URL: https://github.com/apache/airflow/pull/7030#issuecomment-623350479
> For now, I'll make this Executor an optional plugin for Airflow. Thank you
all for contributing and your suggestions.
@aelzeiny Do I understand it correctly that you want to publish it in your
repo rather than apache/airflow? Then we can close this PR, correct? Just want
to be sure so that the stale bot does not close it if you still want to work on
it here :)
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
> AWS Fargate Executor (AIP-29) (WIP)
> ---
>
> Key: AIRFLOW-6440
> URL: https://issues.apache.org/jira/browse/AIRFLOW-6440
> Project: Apache Airflow
> Issue Type: Improvement
> Components: aws, executors
>Affects Versions: 1.10.8
> Environment: AWS Cloud
>Reporter: Ahmed Elzeiny
>Assignee: Ahmed Elzeiny
>Priority: Minor
> Labels: AWS, Executor, autoscaling
> Original Estimate: 336h
> Remaining Estimate: 336h
>
> h1. Links
> AIP -
> [https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-29%3A+AWS+Fargate+Executor]
> PR - [https://github.com/apache/airflow/pull/7030]
> h1. Airflow on AWS Fargate
> {color:#707070}We propose the creation of a new Airflow Executor, called the
> FargateExecutor, that runs tasks asynchronously on AWS Fargate. The Airflow
> Scheduler comes up with a command that needs to be executed in some shell. A
> Docker container parameterized with the command is passed in as an ARG, and
> AWS Fargate provisions a new instance with . The container then completes or
> fails the job, causing the container to die along with the Fargate instance.
> The executor is responsible for keeping track what happened to the task with
> an airflow task id and AWS ARN number, and based off of the instance exit
> code we either say that the task succeeded or failed.{color}
> h1. Proposed Implementation
> As you could probably deduce, the underlying mechanism to launch, track, and
> stop Fargate instances is AWS' Boto3 Library.
> To accomplish this we create a FargateExecutor under the "airflow.executors"
> module. This class will extend from BaseExecutor and override 5 methods:
> {{start()}}, {{{color:#3366ff}sync(){color}}},{{{color:#3366ff}
> execute_async(){color}}}, {{{color:#3366ff}end(){color}}}, and
> {{{color:#3366ff}terminate(){color}}}. Internally, the FargateExecutor uses
> boto3 for monitoring and deployment purposes.
> {color:#707070}The three major Boto3 API calls are:{color}
> * {color:#707070}The {color:#0747a6}{{execute_async()}}{color} function
> calls boto3's {{{color:#0747a6}run_task(){color}}} function.{color}
> * {color:#707070}The {{{color:#0747a6}sync(){color}}} function calls boto3's
> {{{color:#0747a6}describe_tasks(){color}}} function.{color}
> * {color:#707070}The {{{color:#0747a6}terminate(){color}}} function calls
> boto3's {{{color:#0747a6}stop_task(){color}}} function.{color}
> h1. Maintenance
> The executor itself is nothing special since it mostly relies on overriding
> the proper methods from .
> In general, AWS is fairly committed to keeping their APIs in service. Fargate
> is rather new and I've personally perceived a lot more features added as
> optional parameters over the course of the past year. However, the required
> parameters for the three Boto3 calls that are used have remained the same.
> I've also written test-cases that ensures that the Boto3 calls made are
> complaint to the most current version of their APIs.
> We've also introduced a callback hook (very similar to the Celery Executor)
> that allows users to launch tasks with their own parameters. Therefore if a
> user doesn't like the default parameter options used in Boto3's
> \{{run_task(),}}then they can call it themselves with whatever parameters
> they want. This means that Airflow doesn't have to add a new configuration
> everytime AWS makes an addition to AWS Fargate. It's just one configuration
> to cover them all.
> h1. {color:#707070}Proposed Configuration{color}
>
> {code:java}
> [fargate]
> # For more information on any of these execution parameters, see the link
> below:
> #
> https://boto3.amazonaws.com/v1/documentation/api/latest/reference/services/ecs.html#ECS.Client.run_task
> # For boto3 credential management, see
> #
>
[jira] [Commented] (AIRFLOW-6440) AWS Fargate Executor (AIP-29) (WIP)
[
https://issues.apache.org/jira/browse/AIRFLOW-6440?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17098794#comment-17098794
]
ASF GitHub Bot commented on AIRFLOW-6440:
-
feluelle commented on pull request #7030:
URL: https://github.com/apache/airflow/pull/7030#issuecomment-623350479
> For now, I'll make this Executor an optional plugin for Airflow. Thank you
all for contributing and your suggestions.
Do I understand it correctly that you want to publish it in your repo rather
than apache/airflow? Then we can close this PR, correct? Just want to be sure
so that the stale bot does not close it if you still want to work on it here :)
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
> AWS Fargate Executor (AIP-29) (WIP)
> ---
>
> Key: AIRFLOW-6440
> URL: https://issues.apache.org/jira/browse/AIRFLOW-6440
> Project: Apache Airflow
> Issue Type: Improvement
> Components: aws, executors
>Affects Versions: 1.10.8
> Environment: AWS Cloud
>Reporter: Ahmed Elzeiny
>Assignee: Ahmed Elzeiny
>Priority: Minor
> Labels: AWS, Executor, autoscaling
> Original Estimate: 336h
> Remaining Estimate: 336h
>
> h1. Links
> AIP -
> [https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-29%3A+AWS+Fargate+Executor]
> PR - [https://github.com/apache/airflow/pull/7030]
> h1. Airflow on AWS Fargate
> {color:#707070}We propose the creation of a new Airflow Executor, called the
> FargateExecutor, that runs tasks asynchronously on AWS Fargate. The Airflow
> Scheduler comes up with a command that needs to be executed in some shell. A
> Docker container parameterized with the command is passed in as an ARG, and
> AWS Fargate provisions a new instance with . The container then completes or
> fails the job, causing the container to die along with the Fargate instance.
> The executor is responsible for keeping track what happened to the task with
> an airflow task id and AWS ARN number, and based off of the instance exit
> code we either say that the task succeeded or failed.{color}
> h1. Proposed Implementation
> As you could probably deduce, the underlying mechanism to launch, track, and
> stop Fargate instances is AWS' Boto3 Library.
> To accomplish this we create a FargateExecutor under the "airflow.executors"
> module. This class will extend from BaseExecutor and override 5 methods:
> {{start()}}, {{{color:#3366ff}sync(){color}}},{{{color:#3366ff}
> execute_async(){color}}}, {{{color:#3366ff}end(){color}}}, and
> {{{color:#3366ff}terminate(){color}}}. Internally, the FargateExecutor uses
> boto3 for monitoring and deployment purposes.
> {color:#707070}The three major Boto3 API calls are:{color}
> * {color:#707070}The {color:#0747a6}{{execute_async()}}{color} function
> calls boto3's {{{color:#0747a6}run_task(){color}}} function.{color}
> * {color:#707070}The {{{color:#0747a6}sync(){color}}} function calls boto3's
> {{{color:#0747a6}describe_tasks(){color}}} function.{color}
> * {color:#707070}The {{{color:#0747a6}terminate(){color}}} function calls
> boto3's {{{color:#0747a6}stop_task(){color}}} function.{color}
> h1. Maintenance
> The executor itself is nothing special since it mostly relies on overriding
> the proper methods from .
> In general, AWS is fairly committed to keeping their APIs in service. Fargate
> is rather new and I've personally perceived a lot more features added as
> optional parameters over the course of the past year. However, the required
> parameters for the three Boto3 calls that are used have remained the same.
> I've also written test-cases that ensures that the Boto3 calls made are
> complaint to the most current version of their APIs.
> We've also introduced a callback hook (very similar to the Celery Executor)
> that allows users to launch tasks with their own parameters. Therefore if a
> user doesn't like the default parameter options used in Boto3's
> \{{run_task(),}}then they can call it themselves with whatever parameters
> they want. This means that Airflow doesn't have to add a new configuration
> everytime AWS makes an addition to AWS Fargate. It's just one configuration
> to cover them all.
> h1. {color:#707070}Proposed Configuration{color}
>
> {code:java}
> [fargate]
> # For more information on any of these execution parameters, see the link
> below:
> #
> https://boto3.amazonaws.com/v1/documentation/api/latest/reference/services/ecs.html#ECS.Client.run_task
> # For boto3 credential management, see
> #
>
[jira] [Commented] (AIRFLOW-6440) AWS Fargate Executor (AIP-29) (WIP)
[
https://issues.apache.org/jira/browse/AIRFLOW-6440?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17098551#comment-17098551
]
ASF GitHub Bot commented on AIRFLOW-6440:
-
stale[bot] commented on pull request #7030:
URL: https://github.com/apache/airflow/pull/7030#issuecomment-623172002
This issue has been automatically marked as stale because it has not had
recent activity. It will be closed if no further activity occurs. Thank you for
your contributions.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
> AWS Fargate Executor (AIP-29) (WIP)
> ---
>
> Key: AIRFLOW-6440
> URL: https://issues.apache.org/jira/browse/AIRFLOW-6440
> Project: Apache Airflow
> Issue Type: Improvement
> Components: aws, executors
>Affects Versions: 1.10.8
> Environment: AWS Cloud
>Reporter: Ahmed Elzeiny
>Assignee: Ahmed Elzeiny
>Priority: Minor
> Labels: AWS, Executor, autoscaling
> Original Estimate: 336h
> Remaining Estimate: 336h
>
> h1. Links
> AIP -
> [https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-29%3A+AWS+Fargate+Executor]
> PR - [https://github.com/apache/airflow/pull/7030]
> h1. Airflow on AWS Fargate
> {color:#707070}We propose the creation of a new Airflow Executor, called the
> FargateExecutor, that runs tasks asynchronously on AWS Fargate. The Airflow
> Scheduler comes up with a command that needs to be executed in some shell. A
> Docker container parameterized with the command is passed in as an ARG, and
> AWS Fargate provisions a new instance with . The container then completes or
> fails the job, causing the container to die along with the Fargate instance.
> The executor is responsible for keeping track what happened to the task with
> an airflow task id and AWS ARN number, and based off of the instance exit
> code we either say that the task succeeded or failed.{color}
> h1. Proposed Implementation
> As you could probably deduce, the underlying mechanism to launch, track, and
> stop Fargate instances is AWS' Boto3 Library.
> To accomplish this we create a FargateExecutor under the "airflow.executors"
> module. This class will extend from BaseExecutor and override 5 methods:
> {{start()}}, {{{color:#3366ff}sync(){color}}},{{{color:#3366ff}
> execute_async(){color}}}, {{{color:#3366ff}end(){color}}}, and
> {{{color:#3366ff}terminate(){color}}}. Internally, the FargateExecutor uses
> boto3 for monitoring and deployment purposes.
> {color:#707070}The three major Boto3 API calls are:{color}
> * {color:#707070}The {color:#0747a6}{{execute_async()}}{color} function
> calls boto3's {{{color:#0747a6}run_task(){color}}} function.{color}
> * {color:#707070}The {{{color:#0747a6}sync(){color}}} function calls boto3's
> {{{color:#0747a6}describe_tasks(){color}}} function.{color}
> * {color:#707070}The {{{color:#0747a6}terminate(){color}}} function calls
> boto3's {{{color:#0747a6}stop_task(){color}}} function.{color}
> h1. Maintenance
> The executor itself is nothing special since it mostly relies on overriding
> the proper methods from .
> In general, AWS is fairly committed to keeping their APIs in service. Fargate
> is rather new and I've personally perceived a lot more features added as
> optional parameters over the course of the past year. However, the required
> parameters for the three Boto3 calls that are used have remained the same.
> I've also written test-cases that ensures that the Boto3 calls made are
> complaint to the most current version of their APIs.
> We've also introduced a callback hook (very similar to the Celery Executor)
> that allows users to launch tasks with their own parameters. Therefore if a
> user doesn't like the default parameter options used in Boto3's
> \{{run_task(),}}then they can call it themselves with whatever parameters
> they want. This means that Airflow doesn't have to add a new configuration
> everytime AWS makes an addition to AWS Fargate. It's just one configuration
> to cover them all.
> h1. {color:#707070}Proposed Configuration{color}
>
> {code:java}
> [fargate]
> # For more information on any of these execution parameters, see the link
> below:
> #
> https://boto3.amazonaws.com/v1/documentation/api/latest/reference/services/ecs.html#ECS.Client.run_task
> # For boto3 credential management, see
> #
> https://boto3.amazonaws.com/v1/documentation/api/latest/guide/configuration.html
> ### MANDATORY CONFIGS:
> # Name of region
> region = us-west-2
> # Name of cluster
> cluster = test-airflow
> ### EITHER POPULATE THESE:
> # Name
[jira] [Commented] (AIRFLOW-6440) AWS Fargate Executor (AIP-29) (WIP)
[
https://issues.apache.org/jira/browse/AIRFLOW-6440?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17050328#comment-17050328
]
Ahmed Elzeiny commented on AIRFLOW-6440:
Update 03/03/2020:
Since AWS ECS/Fargate is a proprietary technology, it's hard to maintain the
breeze environment and develop integration tests. There are very valid concerns
about maintenance going forward if we lack an AWS ECS cluster to test on. After
2 months of hard work, this executor is a hard-pass.
It's also true that you can scale the Celery Executor on ECS or Fargate, and
you would have a better time doing so. As an added bonus there would be 0 code
involved. As of December, this is possible because Fargate added EFS support.
I'm currently working on the cloud-formation stack that would spin this up.
Effectively, The AWS Scheduler would put messages in an SQS which is monitored
through CloudWatch which is hooked into an Application-AutoScaler for the AWS
Service which triggers a Capacity Provider. AWS technically gives you the
tools; it's just hard to string it all together.
> AWS Fargate Executor (AIP-29) (WIP)
> ---
>
> Key: AIRFLOW-6440
> URL: https://issues.apache.org/jira/browse/AIRFLOW-6440
> Project: Apache Airflow
> Issue Type: Improvement
> Components: aws, executors
>Affects Versions: 1.10.8
> Environment: AWS Cloud
>Reporter: Ahmed Elzeiny
>Assignee: Ahmed Elzeiny
>Priority: Minor
> Labels: AWS, Executor, autoscaling
> Original Estimate: 336h
> Remaining Estimate: 336h
>
> h1. Links
> AIP -
> [https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-29%3A+AWS+Fargate+Executor]
> PR - [https://github.com/apache/airflow/pull/7030]
> h1. Airflow on AWS Fargate
> {color:#707070}We propose the creation of a new Airflow Executor, called the
> FargateExecutor, that runs tasks asynchronously on AWS Fargate. The Airflow
> Scheduler comes up with a command that needs to be executed in some shell. A
> Docker container parameterized with the command is passed in as an ARG, and
> AWS Fargate provisions a new instance with . The container then completes or
> fails the job, causing the container to die along with the Fargate instance.
> The executor is responsible for keeping track what happened to the task with
> an airflow task id and AWS ARN number, and based off of the instance exit
> code we either say that the task succeeded or failed.{color}
> h1. Proposed Implementation
> As you could probably deduce, the underlying mechanism to launch, track, and
> stop Fargate instances is AWS' Boto3 Library.
> To accomplish this we create a FargateExecutor under the "airflow.executors"
> module. This class will extend from BaseExecutor and override 5 methods:
> {{start()}}, {{{color:#3366ff}sync(){color}}},{{{color:#3366ff}
> execute_async(){color}}}, {{{color:#3366ff}end(){color}}}, and
> {{{color:#3366ff}terminate(){color}}}. Internally, the FargateExecutor uses
> boto3 for monitoring and deployment purposes.
> {color:#707070}The three major Boto3 API calls are:{color}
> * {color:#707070}The {color:#0747a6}{{execute_async()}}{color} function
> calls boto3's {{{color:#0747a6}run_task(){color}}} function.{color}
> * {color:#707070}The {{{color:#0747a6}sync(){color}}} function calls boto3's
> {{{color:#0747a6}describe_tasks(){color}}} function.{color}
> * {color:#707070}The {{{color:#0747a6}terminate(){color}}} function calls
> boto3's {{{color:#0747a6}stop_task(){color}}} function.{color}
> h1. Maintenance
> The executor itself is nothing special since it mostly relies on overriding
> the proper methods from .
> In general, AWS is fairly committed to keeping their APIs in service. Fargate
> is rather new and I've personally perceived a lot more features added as
> optional parameters over the course of the past year. However, the required
> parameters for the three Boto3 calls that are used have remained the same.
> I've also written test-cases that ensures that the Boto3 calls made are
> complaint to the most current version of their APIs.
> We've also introduced a callback hook (very similar to the Celery Executor)
> that allows users to launch tasks with their own parameters. Therefore if a
> user doesn't like the default parameter options used in Boto3's
> \{{run_task(),}}then they can call it themselves with whatever parameters
> they want. This means that Airflow doesn't have to add a new configuration
> everytime AWS makes an addition to AWS Fargate. It's just one configuration
> to cover them all.
> h1. {color:#707070}Proposed Configuration{color}
>
> {code:java}
> [fargate]
> # For more information on any of these execution parameters, see the link
> below:
> #
>
[jira] [Commented] (AIRFLOW-6440) AWS Fargate Executor (AIP-29) (WIP)
[
https://issues.apache.org/jira/browse/AIRFLOW-6440?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17050320#comment-17050320
]
Ahmed Elzeiny commented on AIRFLOW-6440:
Hey Andrea,
You're not wrong. I've made the suggested change.
> AWS Fargate Executor (AIP-29) (WIP)
> ---
>
> Key: AIRFLOW-6440
> URL: https://issues.apache.org/jira/browse/AIRFLOW-6440
> Project: Apache Airflow
> Issue Type: Improvement
> Components: aws, executors
>Affects Versions: 1.10.8
> Environment: AWS Cloud
>Reporter: Ahmed Elzeiny
>Assignee: Ahmed Elzeiny
>Priority: Minor
> Labels: AWS, Executor, autoscaling
> Original Estimate: 336h
> Remaining Estimate: 336h
>
> h1. Links
> AIP -
> [https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-29%3A+AWS+Fargate+Executor]
> PR - [https://github.com/apache/airflow/pull/7030]
> h1. Airflow on AWS Fargate
> {color:#707070}We propose the creation of a new Airflow Executor, called the
> FargateExecutor, that runs tasks asynchronously on AWS Fargate. The Airflow
> Scheduler comes up with a command that needs to be executed in some shell. A
> Docker container parameterized with the command is passed in as an ARG, and
> AWS Fargate provisions a new instance with . The container then completes or
> fails the job, causing the container to die along with the Fargate instance.
> The executor is responsible for keeping track what happened to the task with
> an airflow task id and AWS ARN number, and based off of the instance exit
> code we either say that the task succeeded or failed.{color}
> h1. Proposed Implementation
> As you could probably deduce, the underlying mechanism to launch, track, and
> stop Fargate instances is AWS' Boto3 Library.
> To accomplish this we create a FargateExecutor under the "airflow.executors"
> module. This class will extend from BaseExecutor and override 5 methods:
> {{start()}}, {{{color:#3366ff}sync(){color}}},{{{color:#3366ff}
> execute_async(){color}}}, {{{color:#3366ff}end(){color}}}, and
> {{{color:#3366ff}terminate(){color}}}. Internally, the FargateExecutor uses
> boto3 for monitoring and deployment purposes.
> {color:#707070}The three major Boto3 API calls are:{color}
> * {color:#707070}The {color:#0747a6}{{execute_async()}}{color} function
> calls boto3's {{{color:#0747a6}run_task(){color}}} function.{color}
> * {color:#707070}The {{{color:#0747a6}sync(){color}}} function calls boto3's
> {{{color:#0747a6}describe_tasks(){color}}} function.{color}
> * {color:#707070}The {{{color:#0747a6}terminate(){color}}} function calls
> boto3's {{{color:#0747a6}stop_task(){color}}} function.{color}
> h1. Maintenance
> The executor itself is nothing special since it mostly relies on overriding
> the proper methods from .
> In general, AWS is fairly committed to keeping their APIs in service. Fargate
> is rather new and I've personally perceived a lot more features added as
> optional parameters over the course of the past year. However, the required
> parameters for the three Boto3 calls that are used have remained the same.
> I've also written test-cases that ensures that the Boto3 calls made are
> complaint to the most current version of their APIs.
> We've also introduced a callback hook (very similar to the Celery Executor)
> that allows users to launch tasks with their own parameters. Therefore if a
> user doesn't like the default parameter options used in Boto3's
> \{{run_task(),}}then they can call it themselves with whatever parameters
> they want. This means that Airflow doesn't have to add a new configuration
> everytime AWS makes an addition to AWS Fargate. It's just one configuration
> to cover them all.
> h1. {color:#707070}Proposed Configuration{color}
>
> {code:java}
> [fargate]
> # For more information on any of these execution parameters, see the link
> below:
> #
> https://boto3.amazonaws.com/v1/documentation/api/latest/reference/services/ecs.html#ECS.Client.run_task
> # For boto3 credential management, see
> #
> https://boto3.amazonaws.com/v1/documentation/api/latest/guide/configuration.html
> ### MANDATORY CONFIGS:
> # Name of region
> region = us-west-2
> # Name of cluster
> cluster = test-airflow
> ### EITHER POPULATE THESE:
> # Name of task definition with a bootable-container. Note that this container
> will receive an airflow CLI
> # command as an additional parameter to its entrypoint. It's job is to
> boot-up and run this command
> task_definition = test-airflow-worker
> # name of registered container within your AWS cluster
> container_name = airflow-worker
> # security group ids for task to run in (comma-separated)
> security_groups = sg-xx
> # Subnets for task to run in.
> subnets = subnet-yy,subnet-z
> # FARGATE platform version. Defaults to
[jira] [Commented] (AIRFLOW-6440) AWS Fargate Executor (AIP-29) (WIP)
[
https://issues.apache.org/jira/browse/AIRFLOW-6440?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17027879#comment-17027879
]
Andrea Falco commented on AIRFLOW-6440:
---
I've a question: why is this executor called "AWS Fargate" and not "AWS ECS"?
In practice it allow you to start a task with ECS placed into EC2 or FARGATE
(or FARGATE-SPOT).
I propose to rename this executor with the name "AWS ECS Executor" and correct
the code because in some parte you use the term ECS referring to EC2 and the
term FARGATE referring to ECS.
Tell me if I can help.
> AWS Fargate Executor (AIP-29) (WIP)
> ---
>
> Key: AIRFLOW-6440
> URL: https://issues.apache.org/jira/browse/AIRFLOW-6440
> Project: Apache Airflow
> Issue Type: Improvement
> Components: aws, executors
>Affects Versions: 1.10.8
> Environment: AWS Cloud
>Reporter: Ahmed Elzeiny
>Assignee: Ahmed Elzeiny
>Priority: Minor
> Labels: AWS, Executor, autoscaling
> Original Estimate: 336h
> Remaining Estimate: 336h
>
> h1. Links
> AIP -
> [https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-29%3A+AWS+Fargate+Executor]
> PR - [https://github.com/apache/airflow/pull/7030]
> h1. Airflow on AWS Fargate
> {color:#707070}We propose the creation of a new Airflow Executor, called the
> FargateExecutor, that runs tasks asynchronously on AWS Fargate. The Airflow
> Scheduler comes up with a command that needs to be executed in some shell. A
> Docker container parameterized with the command is passed in as an ARG, and
> AWS Fargate provisions a new instance with . The container then completes or
> fails the job, causing the container to die along with the Fargate instance.
> The executor is responsible for keeping track what happened to the task with
> an airflow task id and AWS ARN number, and based off of the instance exit
> code we either say that the task succeeded or failed.{color}
> h1. Proposed Implementation
> As you could probably deduce, the underlying mechanism to launch, track, and
> stop Fargate instances is AWS' Boto3 Library.
> To accomplish this we create a FargateExecutor under the "airflow.executors"
> module. This class will extend from BaseExecutor and override 5 methods:
> {{start()}}, {{{color:#3366ff}sync(){color}}},{{{color:#3366ff}
> execute_async(){color}}}, {{{color:#3366ff}end(){color}}}, and
> {{{color:#3366ff}terminate(){color}}}. Internally, the FargateExecutor uses
> boto3 for monitoring and deployment purposes.
> {color:#707070}The three major Boto3 API calls are:{color}
> * {color:#707070}The {color:#0747a6}{{execute_async()}}{color} function
> calls boto3's {{{color:#0747a6}run_task(){color}}} function.{color}
> * {color:#707070}The {{{color:#0747a6}sync(){color}}} function calls boto3's
> {{{color:#0747a6}describe_tasks(){color}}} function.{color}
> * {color:#707070}The {{{color:#0747a6}terminate(){color}}} function calls
> boto3's {{{color:#0747a6}stop_task(){color}}} function.{color}
> h1. Maintenance
> The executor itself is nothing special since it mostly relies on overriding
> the proper methods from .
> In general, AWS is fairly committed to keeping their APIs in service. Fargate
> is rather new and I've personally perceived a lot more features added as
> optional parameters over the course of the past year. However, the required
> parameters for the three Boto3 calls that are used have remained the same.
> I've also written test-cases that ensures that the Boto3 calls made are
> complaint to the most current version of their APIs.
> We've also introduced a callback hook (very similar to the Celery Executor)
> that allows users to launch tasks with their own parameters. Therefore if a
> user doesn't like the default parameter options used in Boto3's
> \{{run_task(),}}then they can call it themselves with whatever parameters
> they want. This means that Airflow doesn't have to add a new configuration
> everytime AWS makes an addition to AWS Fargate. It's just one configuration
> to cover them all.
> h1. {color:#707070}Proposed Configuration{color}
>
> {code:java}
> [fargate]
> # For more information on any of these execution parameters, see the link
> below:
> #
> https://boto3.amazonaws.com/v1/documentation/api/latest/reference/services/ecs.html#ECS.Client.run_task
> # For boto3 credential management, see
> #
> https://boto3.amazonaws.com/v1/documentation/api/latest/guide/configuration.html
> ### MANDATORY CONFIGS:
> # Name of region
> region = us-west-2
> # Name of cluster
> cluster = test-airflow
> ### EITHER POPULATE THESE:
> # Name of task definition with a bootable-container. Note that this container
> will receive an airflow CLI
> # command as an additional parameter to its entrypoint. It's job is to
> boot-up and run this command
> task_definition =
[jira] [Commented] (AIRFLOW-6440) AWS Fargate Executor (AIP-29) (WIP)
[
https://issues.apache.org/jira/browse/AIRFLOW-6440?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17007799#comment-17007799
]
ASF GitHub Bot commented on AIRFLOW-6440:
-
aelzeiny commented on pull request #7030: [AIRFLOW-6440][AIP-29] Add AWS
Fargate Executor
URL: https://github.com/apache/airflow/pull/7030
---
Link to JIRA issue: https://issues.apache.org/jira/browse/AIRFLOW-6440
- [X ] Description above provides context of the change
- [X] Commit message starts with `[AIRFLOW-]`, where AIRFLOW- = JIRA
ID*
- [X] Unit tests coverage for changes (not needed for documentation changes)
- [X] Commits follow "[How to write a good git commit
message](http://chris.beams.io/posts/git-commit/)"
- [] Relevant documentation is updated including usage instructions.
- [X] I will engage committers as explained in [Contribution Workflow
Example](https://github.com/apache/airflow/blob/master/CONTRIBUTING.rst#contribution-workflow-example).
(*) For document-only changes, no JIRA issue is needed. Commit message
starts `[AIRFLOW-]`.
---
In case of fundamental code change, Airflow Improvement Proposal
([AIP](https://cwiki.apache.org/confluence/display/AIRFLOW/Airflow+Improvements+Proposals))
is needed.
In case of a new dependency, check compliance with the [ASF 3rd Party
License Policy](https://www.apache.org/legal/resolved.html#category-x).
In case of backwards incompatible changes please leave a note in
[UPDATING.md](https://github.com/apache/airflow/blob/master/UPDATING.md).
Read the [Pull Request
Guidelines](https://github.com/apache/airflow/blob/master/CONTRIBUTING.rst#pull-request-guidelines)
for more information.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
> AWS Fargate Executor (AIP-29) (WIP)
> ---
>
> Key: AIRFLOW-6440
> URL: https://issues.apache.org/jira/browse/AIRFLOW-6440
> Project: Apache Airflow
> Issue Type: Improvement
> Components: aws, executors
>Affects Versions: 1.10.8
> Environment: AWS Cloud
>Reporter: Ahmed Elzeiny
>Assignee: Ahmed Elzeiny
>Priority: Minor
> Labels: AWS, Executor, autoscaling
> Original Estimate: 336h
> Remaining Estimate: 336h
>
> h1. AIP
> [https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-29%3A+AWS+Fargate+Executor]
> h1. Airflow on AWS Fargate
> {color:#707070}We propose the creation of a new Airflow Executor, called the
> FargateExecutor, that runs tasks asynchronously on AWS Fargate. The Airflow
> Scheduler comes up with a command that needs to be executed in some shell. A
> Docker container parameterized with the command is passed in as an ARG, and
> AWS Fargate provisions a new instance with . The container then completes or
> fails the job, causing the container to die along with the Fargate instance.
> The executor is responsible for keeping track what happened to the task with
> an airflow task id and AWS ARN number, and based off of the instance exit
> code we either say that the task succeeded or failed.{color}
> h1. Proposed Implementation
> As you could probably deduce, the underlying mechanism to launch, track, and
> stop Fargate instances is AWS' Boto3 Library.
> To accomplish this we create a FargateExecutor under the "airflow.executors"
> module. This class will extend from BaseExecutor and override 5 methods:
> {color:#0747a6}start(){color},
> {{{color:#3366ff}sync(){color}}},{{{color:#3366ff} execute_async(){color}}},
> {{{color:#3366ff}end(){color}}}, and {{{color:#3366ff}terminate(){color}}}.
> Internally, the FargateExecutor uses boto3 for monitoring and deployment
> purposes.
> {color:#707070}The three major Boto3 API calls are:{color}
> * The {color:#0747a6}{{execute_async()}}{color} function calls boto3's
> {color:#3366ff}{{run_task()}}{color} function.
> * {color:#707070} The
> {{{color:#0747a6}sync{color}}}{color}{{{color:#0747a6}(){color}}} function
> calls boto3's {{{color:#3366ff}describe_tasks(){color}}} function.
> * {color:#707070}The
> {{{color:#0747a6}terminate{color}}}{color}{{{color:#0747a6}(){color}}}
> function calls boto3's {{{color:#3366ff}stop_task(){color}}} function.
> h1. Maintenance
> The executor itself is nothing special since it mostly relies on overriding
> the proper methods from .
> In general, AWS is fairly committed to keeping their APIs in service. Fargate
> is rather new and I've personally perceived a lot more features added as
> optional parameters over the course of the past year. However, the
