o-nikolas commented on code in PR #34381: URL: https://github.com/apache/airflow/pull/34381#discussion_r1327609295
########## airflow/providers/amazon/aws/executors/ecs/Dockerfile: ########## @@ -0,0 +1,86 @@ +# hadolint ignore=DL3007 +FROM apache/airflow:latest +USER root +RUN apt-get update \ + && apt-get install -y --no-install-recommends unzip \ + # The below helps to keep the image size down + && apt-get clean \ + && rm -rf /var/lib/apt/lists/* +RUN curl "https://awscli.amazonaws.com/awscli-exe-linux-x86_64.zip"; -o "awscliv2.zip" +RUN unzip awscliv2.zip && ./aws/install + +# Add a script to run the aws s3 sync command when the container is run +COPY <<"EOF" /entrypoint.sh +#!/bin/bash + +echo "Downloading DAGs from S3 bucket" +aws s3 sync "$S3_URL" "$CONTAINER_DAG_PATH" + +exec "$@" +EOF + +RUN chmod +x /entrypoint.sh + +USER airflow + +## Installing Python Dependencies +# Python dependencies can be installed by providing a requirements.txt. +# If the file is in a different location, use the requirements_path build argument to specify +# the file path. +ARG requirements_path=./requirements.txt +ENV REQUIREMENTS_PATH=$requirements_path + +# Uncomment the two lines below to copy the requirements.txt file to the container, and +# install the dependencies. +# COPY --chown=airflow:root $REQUIREMENTS_PATH /opt/airflow/requirements.txt +# RUN pip install --no-cache-dir -r /opt/airflow/requirements.txt + + +## AWS Authentication +# The image requires access to AWS services. This Dockerfile supports 2 ways to authenticate with AWS. +# The first is using build arguments where you can provide the AWS credentials as arguments +# passed when building the image. The other option is to copy the ~/.aws folder to the container, +# and authenticate using the credentials in that folder. +# If you would like to use an alternative method of authentication, feel free to make the +# necessary changes to this file. + +# Use these arguments to provide AWS authentication information +ARG aws_access_key_id +ARG aws_secret_access_key +ARG aws_default_region +ARG aws_session_token + +ENV AWS_ACCESS_KEY_ID=$aws_access_key_id Review Comment: There is a `conn_id` that can be used with the ECS executor, which I think is the most proper way to do it, but it's a bit more work and requires the user to go configure that. Like I mentioned above in another thread, this image is really just a very simple example to get people started. If folks are using this executor in production, they likely will have their own preferred and robust mechanism for managing credentials in their images. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
