Taragolis commented on code in PR #34381: URL: https://github.com/apache/airflow/pull/34381#discussion_r1326653046
########## airflow/providers/amazon/aws/config_templates/config.yml: ########## @@ -0,0 +1,131 @@ +# Licensed to the Apache Software Foundation (ASF) under one +# or more contributor license agreements. See the NOTICE file +# distributed with this work for additional information +# regarding copyright ownership. The ASF licenses this file +# to you under the Apache License, Version 2.0 (the +# "License"); you may not use this file except in compliance +# with the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, +# software distributed under the License is distributed on an +# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +# KIND, either express or implied. See the License for the +# specific language governing permissions and limitations +# under the License. + +--- + +aws_ecs_executor: + description: | + This section only applies if you are using the AwsEcsExecutor in + Airflow's ``[core]`` configuration. + For more information on any of these execution parameters, see the link below: + https://boto3.amazonaws.com/v1/documentation/api/latest/reference/services/ecs/client/run_task.html + For boto3 credential management, see + https://boto3.amazonaws.com/v1/documentation/api/latest/guide/configuration.html + options: + conn_id: + description: | + The Airflow connection (i.e. credentials) used by the ECS executor to make API calls to AWS ECS. + version_added: "2.8" + type: string + example: "aws_default" + default: "aws_default" Review Comment: Maybe better set default to `None`, so in this case it will explicitly use boto3 creds strategy, e.g. ECS Task Role / Execution Task Role? ########## airflow/providers/amazon/aws/executors/ecs/Dockerfile: ########## @@ -0,0 +1,86 @@ +# hadolint ignore=DL3007 +FROM apache/airflow:latest +USER root +RUN apt-get update \ + && apt-get install -y --no-install-recommends unzip \ + # The below helps to keep the image size down + && apt-get clean \ + && rm -rf /var/lib/apt/lists/* +RUN curl "https://awscli.amazonaws.com/awscli-exe-linux-x86_64.zip"; -o "awscliv2.zip" +RUN unzip awscliv2.zip && ./aws/install Review Comment: I just think about internal airflow ci script 🤔 Which might covered most common problem with installing aws-cli into the docker https://github.com/apache/airflow/blob/main/scripts/in_container/bin/install_aws.sh ########## airflow/providers/amazon/aws/executors/ecs/Dockerfile: ########## @@ -0,0 +1,86 @@ +# hadolint ignore=DL3007 +FROM apache/airflow:latest +USER root +RUN apt-get update \ + && apt-get install -y --no-install-recommends unzip \ + # The below helps to keep the image size down + && apt-get clean \ + && rm -rf /var/lib/apt/lists/* +RUN curl "https://awscli.amazonaws.com/awscli-exe-linux-x86_64.zip"; -o "awscliv2.zip" +RUN unzip awscliv2.zip && ./aws/install + +# Add a script to run the aws s3 sync command when the container is run +COPY <<"EOF" /entrypoint.sh +#!/bin/bash + +echo "Downloading DAGs from S3 bucket" +aws s3 sync "$S3_URL" "$CONTAINER_DAG_PATH" + +exec "$@" +EOF + +RUN chmod +x /entrypoint.sh + +USER airflow + +## Installing Python Dependencies +# Python dependencies can be installed by providing a requirements.txt. +# If the file is in a different location, use the requirements_path build argument to specify +# the file path. +ARG requirements_path=./requirements.txt +ENV REQUIREMENTS_PATH=$requirements_path + +# Uncomment the two lines below to copy the requirements.txt file to the container, and +# install the dependencies. +# COPY --chown=airflow:root $REQUIREMENTS_PATH /opt/airflow/requirements.txt +# RUN pip install --no-cache-dir -r /opt/airflow/requirements.txt + + +## AWS Authentication +# The image requires access to AWS services. This Dockerfile supports 2 ways to authenticate with AWS. +# The first is using build arguments where you can provide the AWS credentials as arguments +# passed when building the image. The other option is to copy the ~/.aws folder to the container, +# and authenticate using the credentials in that folder. +# If you would like to use an alternative method of authentication, feel free to make the +# necessary changes to this file. + +# Use these arguments to provide AWS authentication information +ARG aws_access_key_id +ARG aws_secret_access_key +ARG aws_default_region +ARG aws_session_token + +ENV AWS_ACCESS_KEY_ID=$aws_access_key_id Review Comment: IMO better use ECS task roles rather then permanent credentials. @o-nikolas you could ping me in slack and I will share my old terraform module for deployment airflow on ECS, it might be useful at least for minimal IAM roles which required for setting up ECS + security groups between DB and EC2 instances. It could be a bit different rather then Fargate ########## airflow/providers/amazon/aws/executors/ecs/Dockerfile: ########## @@ -0,0 +1,86 @@ +# hadolint ignore=DL3007 +FROM apache/airflow:latest +USER root +RUN apt-get update \ + && apt-get install -y --no-install-recommends unzip \ + # The below helps to keep the image size down + && apt-get clean \ + && rm -rf /var/lib/apt/lists/* +RUN curl "https://awscli.amazonaws.com/awscli-exe-linux-x86_64.zip"; -o "awscliv2.zip" +RUN unzip awscliv2.zip && ./aws/install Review Comment: I just think about internal airflow ci script 🤔 Which might covered most common problem with installing aws-cli into the docker https://github.com/apache/airflow/blob/main/scripts/in_container/bin/install_aws.sh -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
