[ https://issues.apache.org/jira/browse/AIRFLOW-6440?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Ahmed Elzeiny updated AIRFLOW-6440: ----------------------------------- Summary: [WIP] AWS Fargate Executor (AIP-29) (was: AWS Fargate Executor (AIP-29)) > [WIP] AWS Fargate Executor (AIP-29) > ----------------------------------- > > Key: AIRFLOW-6440 > URL: https://issues.apache.org/jira/browse/AIRFLOW-6440 > Project: Apache Airflow > Issue Type: Improvement > Components: aws, executors > Affects Versions: 1.10.8 > Environment: AWS Cloud > Reporter: Ahmed Elzeiny > Assignee: Ahmed Elzeiny > Priority: Minor > Labels: AWS, Executor, autoscaling > Original Estimate: 336h > Remaining Estimate: 336h > > h1. AIP > [https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-29%3A+AWS+Fargate+Executor] > h1. Airflow on AWS Fargate > {color:#707070}We propose the creation of a new Airflow Executor, called the > FargateExecutor, that runs tasks asynchronously on AWS Fargate. The Airflow > Scheduler comes up with a command that needs to be executed in some shell. A > Docker container parameterized with the command is passed in as an ARG, and > AWS Fargate provisions a new instance with . The container then completes or > fails the job, causing the container to die along with the Fargate instance. > The executor is responsible for keeping track what happened to the task with > an airflow task id and AWS ARN number, and based off of the instance exit > code we either say that the task succeeded or failed.{color} > h1. Proposed Implementation > > As you could probably deduce, the underlying mechanism to launch, track, and > stop Fargate instances is AWS' Boto3 Library. > > To accomplish this we create a FargateExecutor under the "airflow.executors" > module. This class will extend from BaseExecutor and override 5 methods: > \{{start()}}, {{{color:#3366ff}sync(){color}}},{{{color:#3366ff} > execute_async(){color}}}, {{{color:#3366ff}end(){color}}}, and > {{{color:#3366ff}terminate(){color}}}. Internally, the FargateExecutor uses > boto3 for monitoring and deployment purposes. > {color:#707070}The three major Boto3 API calls are:{color} > * The \{{execute_async()}} function calls boto3's > {color:#3366ff}{{run_task()}}{color} function. > * {color:#707070} The {{sync{color}{color:#3366ff}(){color}}} function calls > boto3's {{{color:#3366ff}describe_tasks(){color}}} function. > * {color:#707070}The {{terminate{color}{color:#3366ff}(){color}}} function > calls boto3's {{{color:#3366ff}stop_task(){color}}} function. > h1. Maintenance > The executor itself is nothing special since it mostly relies on overriding > the proper methods from . > > In general, AWS is fairly committed to keeping their APIs in service. > Fargate is rather new and I've personally perceived a lot more features added > as optional parameters over the course of the past year. However, the > required parameters for the three Boto3 calls that are used have remained the > same. I've also written test-cases that ensures that the Boto3 calls made are > complaint to the most current version of their APIs. > > We've also introduced a callback hook (very similar to the Celery Executor) > that allows users to launch tasks with their own parameters. Therefore if a > user doesn't like the default parameter options used in Boto3's > {{run_task(),}}then they can call it themselves with whatever parameters they > want. This means that Airflow doesn't have to add a new configuration > everytime AWS makes an addition to AWS Fargate. It's just one configuration > to cover them all. > h1. {color:#707070}Proposed Configuration{color} > > {code:java} > [fargate] > # For more information on any of these execution parameters, see the link > below: > # > https://boto3.amazonaws.com/v1/documentation/api/latest/reference/services/ecs.html#ECS.Client.run_task > # For boto3 credential management, see > # > https://boto3.amazonaws.com/v1/documentation/api/latest/guide/configuration.html > ### MANDATORY CONFIGS: > # Name of region > region = us-west-2 > # Name of cluster > cluster = test-airflow > ### EITHER POPULATE THESE: > # Name of task definition with a bootable-container. Note that this container > will receive an airflow CLI > # command as an additional parameter to its entrypoint. It's job is to > boot-up and run this command > task_definition = test-airflow-worker > # name of registered container within your AWS cluster > container_name = airflow-worker > # security group ids for task to run in (comma-separated) > security_groups = sg-xxxxxxxxxxxxxx > # Subnets for task to run in. > subnets = subnet-yyyyyyyyyy,subnet-zzzzzzzzz > # FARGATE platform version. Defaults to Latest. > platform_version = LATEST > # Launch type can either be 'FARGATE' OR 'ECS'. Defaults to Fargate. > launch_type = FARGATE > # Assign public ip can either be 'ENABLED' or 'DISABLED'. Defaults to > 'ENABLED'. > assign_public_ip = DISABLED > ### OR POPULATE THIS: > # This is a function which returns a function. The outer function takes no > arguments, and returns the inner function. > # The inner function takes in an airflow CLI command an outputs a json > compatible with the boto3 run_task API > # linked above. In other words, if you don't like the way I call the fargate > API then call it yourself > execution_config_function = > airflow.executors.fargate_executor.default_task_id_to_fargate_options_function > {code} > > > -- This message was sent by Atlassian Jira (v8.3.4#803005)