feluelle commented on a change in pull request #9129:
URL: https://github.com/apache/airflow/pull/9129#discussion_r435064284
##########
File path: TESTING.rst
##########
@@ -621,12 +621,11 @@ credentials stored in your ``home`` directory. Use this
feature with care as it
visible to anything that you have installed inside the Docker container.
Currently forwarded credentials are:
- * all credentials stored in ``${HOME}/.config`` (for example, GCP
credentials)
- * credentials stored in ``${HOME}/.gsutil`` for ``gsutil`` tool from GCS
* credentials stored in ``${HOME}/.aws``, ``${HOME}/.boto``, and
``${HOME}/.s3`` (for AWS authentication)
+ * credentials stored in ``${HOME}/.azure``(for Azure authentication)
+ * credentials stored in ``${HOME}/.config`` (for example, GCP credentials)
+ * credentials stored in ``${HOME}/.gsutil`` for ``gsutil`` tool from GCS
* credentials stored in ``${HOME}/.docker`` for docker
Review comment:
Should we sort that alphabetically?
##########
File path: Dockerfile.ci
##########
@@ -338,6 +308,24 @@ RUN if [[ -n "${ADDITIONAL_PYTHON_DEPS}" ]]; then \
pip install ${ADDITIONAL_PYTHON_DEPS}; \
fi
+ARG AWSCLI_IMAGE="amazon/aws-cli:latest"
+ENV AWSCLI_IMAGE=${AWSCLI_IMAGE}
+
+ARG AZURECLI_IMAGE="mcr.microsoft.com/azure-cli:latest"
+ENV AZURECLI_IMAGE=${AZURECLI_IMAGE}
+
+ARG GCLOUD_IMAGE="gcr.io/google.com/cloudsdktool/cloud-sdk:latest"
+ENV GCLOUD_IMAGE=${GCLOUD_IMAGE}
+
+RUN echo -e "\
+ alias aws=\"docker run --rm -it -v /root/.aws:/root/.aws ${AWSCLI_IMAGE}\"
\n\
+ alias aws-update=\"docker pull ${AWSCLI_IMAGE}\" \n\
+ alias az=\"docker run --rm -it -v /root/.azure:/root/.azure
${AZURECLI_IMAGE} az\" \n\
+ alias az-update=\"docker pull ${AZURECLI_IMAGE}\" \n\
+ alias gcloud=\"docker run --rm -it -v /root/.config:/root/.config
${GCLOUD_IMAGE} gcloud\" \n\
+ alias gcloud-update=\"docker pull ${GCLOUD_IMAGE}\" \n\
+ " >> ~/.bashrc
+
Review comment:
I really like this change - looks cleaner to have seperate docker
containers running for those.. but I am wondering how terraform will be able to
use those. I am on adding terraform to the CI and I could use [their docker
image](https://hub.docker.com/r/hashicorp/terraform/dockerfile) as well.
##########
File path: Dockerfile.ci
##########
@@ -208,8 +178,8 @@ ENV AIRFLOW_SOURCES=${AIRFLOW_SOURCES}
WORKDIR ${AIRFLOW_SOURCES}
-RUN mkdir -pv ${AIRFLOW_HOME} \
- mkdir -pv ${AIRFLOW_HOME}/dags \
Review comment:
Did that work? 🤔
##########
File path: Dockerfile.ci
##########
@@ -338,6 +308,24 @@ RUN if [[ -n "${ADDITIONAL_PYTHON_DEPS}" ]]; then \
pip install ${ADDITIONAL_PYTHON_DEPS}; \
fi
+ARG AWSCLI_IMAGE="amazon/aws-cli:latest"
+ENV AWSCLI_IMAGE=${AWSCLI_IMAGE}
+
+ARG AZURECLI_IMAGE="mcr.microsoft.com/azure-cli:latest"
+ENV AZURECLI_IMAGE=${AZURECLI_IMAGE}
+
+ARG GCLOUD_IMAGE="gcr.io/google.com/cloudsdktool/cloud-sdk:latest"
+ENV GCLOUD_IMAGE=${GCLOUD_IMAGE}
Review comment:
What is the use-case for having those as `ARG`s and what is the use-case
for having those as `ENV`s?
##########
File path: BREEZE.rst
##########
@@ -231,6 +232,41 @@ from your ``logs`` directory in the Airflow sources, so
all logs created in the
visible in the host as well. Every time you enter the container, the ``logs``
directory is
cleaned so that logs do not accumulate.
+CLI for cloud providers
+=======================
+
+For development convenience we installed simple wrappers for the most common
cloud providers CLIs. Those
+CLIs are not installed when you build or pull the image - they will be
downloaded as docker images
+the first time you attempt to use them. It is downloaded and executed in your
host's docker engine so once
+it is downloaded, it will stay until you remove the downloaded images from
your host container.
+
+For each of those CLI you need to provide credentials. You can do it in one of
the ways:
+
+* You can forward credentials from your host machine when starting breeze with
+ ``--forward-credentials`` flag added when you start ``Breeze``. Just a
watch-out - this makes
+ your host credentials available inside the image, so use it with care.
+
+* If you do not want to forward credentials, you can use traditional
authentication methods provided by
+ each of the tools (for example ``gcloud auth login``) - the credentials are
stored in a configuration
+ directory specific for each CLI tool until you stop breeze session. You will
need to re-authenticate
+ when you restart ``Breeze``.
+
+Those are currently installed CLIs (they are available as aliases to the
docker commands):
+
++-----------------------+----------+-------------------------------------------------+-------------------+
+| Cloud Provider | CLI tool | Docker image
| Configuration dir |
++=======================+==========+=================================================+===================+
+| Amazon Web Services | aws | amazon/aws-cli:latest
| .aws |
++-----------------------+----------+-------------------------------------------------+-------------------+
+| Google Cloud Platform | gcloud | mcr.microsoft.com/azure-cli:latest
| .config/gcloud |
++-----------------------+----------+-------------------------------------------------+-------------------+
+| Microsoft Azure | az |
gcr.io/google.com/cloudsdktool/cloud-sdk:latest | .azure |
++-----------------------+----------+-------------------------------------------------+-------------------+
+
+For each of the CLIs we have also an accompanying ``*-update`` alias (for
example aws-update) which
+will pull the latest image for the tool.
Review comment:
nice!
##########
File path: Dockerfile.ci
##########
@@ -338,6 +308,24 @@ RUN if [[ -n "${ADDITIONAL_PYTHON_DEPS}" ]]; then \
pip install ${ADDITIONAL_PYTHON_DEPS}; \
fi
+ARG AWSCLI_IMAGE="amazon/aws-cli:latest"
+ENV AWSCLI_IMAGE=${AWSCLI_IMAGE}
+
+ARG AZURECLI_IMAGE="mcr.microsoft.com/azure-cli:latest"
+ENV AZURECLI_IMAGE=${AZURECLI_IMAGE}
+
+ARG GCLOUD_IMAGE="gcr.io/google.com/cloudsdktool/cloud-sdk:latest"
+ENV GCLOUD_IMAGE=${GCLOUD_IMAGE}
Review comment:
I don't think that is necessary.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
[email protected]
