o-nikolas opened a new pull request, #34381: URL: https://github.com/apache/airflow/pull/34381
# Overview Introducing the **AWS ECS executor**! Over the past couple months myself, @ferruzzi and @syedahsn have been hard at work on an AWS ECS executor (if you've not seen us as much in the community, this is why :sweat_smile:), based on an initial contribution from @aelzeiny From the headline sentence in the README: > This is an Airflow executor powered by Amazon Elastic Container Service (ECS). Each task that Airflow schedules for execution is run within its own ECS container. This is an initial release with most of the basic functionality in place. We have many future upgrades, CLI commands, and other features planned, so stay tuned! # Review Notes This is certainly a large PR, which in some ways is not ideal. However, it can be difficult to release portions of a component like this. We've tried to scope all the code here to the minimum required for folks to begin using this executor. When reviewing, I don't think it's required to read every single line of code. I have annotated most modules in the diff view with comments explaining the changes. Please take a look at the portions you think are relevant or that you have some experience in. Ultimately, (almost) all the code is scoped to the Amazon provider package and it is a net new component, so there is a very limited blast radius. Very little existing user workflows or code should be affected. Included in the list below are the areas of existing code that have been updated, which have the possibility of affecting user workflows. **Some specific changes to pay particular attention to**: - The new logging environment variable to enable task logs to work correctly in containerized executors. K8s has an approach for this, which we follow quite closely when making it generic. However, we did not convert K8s to this new mechanism to keep the change set minimal and reduce blast radius. - Updates to boto user agent tagging in the Base Aws hook. - A config yaml is now present in the AWS provider, leveraging @potiuk's changes, this is new code, but the first time we're leveraging this system so it's worth reviewing carefully # Testing There is extensive unit testing which has near 100% line coverage in most cases:  We also performed a lot of manual UAT for things such as: - Load testing (500 concurrent tasks, reaching the limit imposed by ECS, see the performance and tuning section of the README) - Different platform versions of ECS Fargate - Multiple ways to provide AWS credentials to the executor (built into the image, using Airflow connection, etc) - Deferrable operators - Dynamic task mapping - Data Driven Scheduling <!-- Licensed to the Apache Software Foundation (ASF) under one or more contributor license agreements. See the NOTICE file distributed with this work for additional information regarding copyright ownership. The ASF licenses this file to you under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at http://www.apache.org/licenses/LICENSE-2.0 Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License. --> <!-- Thank you for contributing! Please make sure that your code changes are covered with tests. And in case of new features or big changes remember to adjust the documentation. Feel free to ping committers for the review! In case of an existing issue, reference it using one of the following: closes: #ISSUE related: #ISSUE How to write a good git commit message: http://chris.beams.io/posts/git-commit/ --> <!-- Please keep an empty line above the dashes. --> --- **^ Add meaningful description above** Read the **[Pull Request Guidelines](https://github.com/apache/airflow/blob/main/CONTRIBUTING.rst#pull-request-guidelines)** for more information. In case of fundamental code changes, an Airflow Improvement Proposal ([AIP](https://cwiki.apache.org/confluence/display/AIRFLOW/Airflow+Improvement+Proposals)) is needed. In case of a new dependency, check compliance with the [ASF 3rd Party License Policy](https://www.apache.org/legal/resolved.html#category-x). In case of backwards incompatible changes please leave a note in a newsfragment file, named `{pr_number}.significant.rst` or `{issue_number}.significant.rst`, in [newsfragments](https://github.com/apache/airflow/tree/main/newsfragments). -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
