fpopic commented on issue #42396:
URL: https://github.com/apache/airflow/issues/42396#issuecomment-2424793223
Adding minimal change to latest setup to reproduce
In `docker-compose.yml` passing host ADC config with write permission (to
refresh host token from within container)
```yaml
x-airflow-common:
&airflow-common
#...
build: .
#image: ...
environment:
&airflow-common-env
# ...
AIRFLOW__LOGGING__LOGGING_LEVEL: 'WARNING'
AIRFLOW_CONN_GOOGLE_CLOUD_DEFAULT: '{ "conn_type":
"google_cloud_platform", "extra": { "scope":
"https://www.googleapis.com/auth/cloud-platform,https://www.googleapis.com/auth/drive,https://www.googleapis.com/auth/bigquery";,
"num_retries": 0 } }'
GOOGLE_CLOUD_PROJECT: <my-project>
volumes: &airflow-common-volumes
# ...
# Use ADC
- ~/.config/gcloud/:/home/airflow/.config/gcloud:rw
...
```
In Dockerfile install google cloud SDK (`gcloud` command) since google base
hook requires gcloud CLI it:
https://github.com/apache/airflow/blob/2.10.2/airflow/providers/google/common/hooks/base_google.py#L638-L669
```Dockerfile
# specifying explicit python 3.9 otherwise gcloud sdk installation complains
FROM apache/airflow:2.10.2-python3.9
USER root
## COPY FROM
https://github.com/apache/airflow/blob/main/docs/docker-stack/docker-images-recipes/gcloud.Dockerfile
ARG CLOUD_SDK_VERSION=322.0.0
ENV GCLOUD_HOME=/home/airflow/google-cloud-sdk
ENV PATH="${GCLOUD_HOME}/bin/:${PATH}"
USER airflow
RUN
DOWNLOAD_URL="https://dl.google.com/dl/cloudsdk/channels/rapid/downloads/google-cloud-sdk-${CLOUD_SDK_VERSION}-linux-x86_64.tar.gz";
\
&& TMP_DIR="$(mktemp -d)" \
&& curl -fL "${DOWNLOAD_URL}" --output
"${TMP_DIR}/google-cloud-sdk.tar.gz" \
&& mkdir -p "${GCLOUD_HOME}" \
&& tar xzf "${TMP_DIR}/google-cloud-sdk.tar.gz" -C "${GCLOUD_HOME}"
--strip-components=1 \
&& "${GCLOUD_HOME}/install.sh" \
--bash-completion=false \
--path-update=false \
--usage-reporting=false \
--additional-components alpha beta kubectl \
--quiet \
&& rm -rf "${TMP_DIR}" \
&& rm -rf "${GCLOUD_HOME}/.install/.backup/" \
&& gcloud --version
# END OF COPY
# pip install airflow has to be run with USER airflow
RUN pip install 'apache-airflow==2.10.2' \
apache-airflow[google_auth] \
apache-airflow-providers-google \
apache-airflow[google] \
apache-airflow-providers-apache-beam
#apache-airflow-providers-hashicorp \
RUN mkdir -p /home/airflow/.config/gcloud/
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
