ExecutionSpeed opened a new issue #16803: URL: https://github.com/apache/airflow/issues/16803
<!-- Welcome to Apache Airflow! For a smooth issue process, try to answer the following questions. Don't worry if they're not all applicable; just try to include what you can :-) If you need to include code snippets or logs, please put them in fenced code blocks. If they're super-long, please use the details tag like <details><summary>super-long log</summary> lots of stuff </details> Please delete these comment blocks before submitting the issue. --> <!-- IMPORTANT!!! PLEASE CHECK "SIMILAR TO X EXISTING ISSUES" OPTION IF VISIBLE NEXT TO "SUBMIT NEW ISSUE" BUTTON!!! PLEASE CHECK IF THIS ISSUE HAS BEEN REPORTED PREVIOUSLY USING SEARCH!!! Please complete the next sections or the issue will be closed. These questions are the first thing we need to know to understand the context. --> **Apache Airflow version**: 2.1.1 -- Docker Image: `apache/airflow:2.1.1-python3.8` **Kubernetes version (if you are using kubernetes)** (use `kubectl version`): Not running on k8s. **Environment**: - **Cloud provider or hardware configuration**: DigitalOcean - **OS** (e.g. from /etc/os-release): Ubuntu - **Kernel** (e.g. `uname -a`): `Linux airflow 5.11.0-22-generic #23-Ubuntu SMP Thu Jun 17 00:34:23 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux` - **Install tools**: Installed via Docker & Docker Compose following instructions from [official docker-compose installation docs](https://airflow.apache.org/docs/apache-airflow/stable/start/docker.html#docker-compose-yaml). - **Others**: N/A **What happened**: <!-- (please include exact error messages if you can) --> **What you expected to happen**: <!-- What do you think went wrong? --> **How to reproduce it**: <!--- As minimally and precisely as possible. Keep in mind we do not have access to your cluster or dags. If you are using kubernetes, please attempt to recreate the issue using minikube or kind. ## Install minikube/kind - Minikube https://minikube.sigs.k8s.io/docs/start/ - Kind https://kind.sigs.k8s.io/docs/user/quick-start/ If this is a UI bug, please provide a screenshot of the bug or a link to a youtube video of the bug in action You can include images using the .md style of  To record a screencast, mac users can use QuickTime and then create an unlisted youtube video with the resulting .mov file. ---> Running with the `DockerOperator` causes the following error: ``` *** Reading local file: /opt/airflow/logs/docker_eod_us_equities/docker_command_sleep/2021-07-04T23:19:48.431208+00:00/2.log [2021-07-04 23:49:53,886] {taskinstance.py:896} INFO - Dependencies all met for <TaskInstance: docker_eod_us_equities.docker_command_sleep 2021-07-04T23:19:48.431208+00:00 [queued]> [2021-07-04 23:49:53,902] {taskinstance.py:896} INFO - Dependencies all met for <TaskInstance: docker_eod_us_equities.docker_command_sleep 2021-07-04T23:19:48.431208+00:00 [queued]> [2021-07-04 23:49:53,903] {taskinstance.py:1087} INFO - -------------------------------------------------------------------------------- [2021-07-04 23:49:53,903] {taskinstance.py:1088} INFO - Starting attempt 2 of 2 [2021-07-04 23:49:53,903] {taskinstance.py:1089} INFO - -------------------------------------------------------------------------------- [2021-07-04 23:49:53,911] {taskinstance.py:1107} INFO - Executing <Task(DockerOperator): docker_command_sleep> on 2021-07-04T23:19:48.431208+00:00 [2021-07-04 23:49:53,914] {standard_task_runner.py:52} INFO - Started process 3583 to run task [2021-07-04 23:49:53,920] {standard_task_runner.py:76} INFO - Running: ['***', 'tasks', 'run', 'docker_eod_us_equities', 'docker_command_sleep', '2021-07-04T23:19:48.431208+00:00', '--job-id', '52', '--pool', 'default_pool', '--raw', '--subdir', 'DAGS_FOLDER/eod_us_equities.py', '--cfg-path', '/tmp/tmp5iz14yg9', '--error-file', '/tmp/tmpleaymjfa'] [2021-07-04 23:49:53,920] {standard_task_runner.py:77} INFO - Job 52: Subtask docker_command_sleep [2021-07-04 23:49:54,016] {logging_mixin.py:104} INFO - Running <TaskInstance: docker_eod_us_equities.docker_command_sleep 2021-07-04T23:19:48.431208+00:00 [running]> on host 0dff7922cb76 [2021-07-04 23:49:54,172] {taskinstance.py:1300} INFO - Exporting the following env vars: AIRFLOW_CTX_DAG_OWNER=*** AIRFLOW_CTX_DAG_ID=docker_eod_us_equities AIRFLOW_CTX_TASK_ID=docker_command_sleep AIRFLOW_CTX_EXECUTION_DATE=2021-07-04T23:19:48.431208+00:00 AIRFLOW_CTX_DAG_RUN_ID=manual__2021-07-04T23:19:48.431208+00:00 [2021-07-04 23:49:54,205] {docker.py:231} INFO - Starting docker container from image alpine [2021-07-04 23:49:54,216] {taskinstance.py:1501} ERROR - Task failed with exception Traceback (most recent call last): File "/home/airflow/.local/lib/python3.8/site-packages/docker/api/client.py", line 268, in _raise_for_status response.raise_for_status() File "/home/airflow/.local/lib/python3.8/site-packages/requests/models.py", line 943, in raise_for_status raise HTTPError(http_error_msg, response=self) requests.exceptions.HTTPError: 400 Client Error: Bad Request for url: http+docker://localhost/v1.30/containers/create?name=docker_command_sleep During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/airflow/.local/lib/python3.8/site-packages/airflow/models/taskinstance.py", line 1157, in _run_raw_task self._prepare_and_execute_task_with_callbacks(context, task) File "/home/airflow/.local/lib/python3.8/site-packages/airflow/models/taskinstance.py", line 1331, in _prepare_and_execute_task_with_callbacks result = self._execute_task(context, task_copy) File "/home/airflow/.local/lib/python3.8/site-packages/airflow/models/taskinstance.py", line 1361, in _execute_task result = task_copy.execute(context=context) File "/home/airflow/.local/lib/python3.8/site-packages/airflow/providers/docker/operators/docker.py", line 319, in execute return self._run_image() File "/home/airflow/.local/lib/python3.8/site-packages/airflow/providers/docker/operators/docker.py", line 237, in _run_image self.container = self.cli.create_container( File "/home/airflow/.local/lib/python3.8/site-packages/docker/api/container.py", line 430, in create_container return self.create_container_from_config(config, name) File "/home/airflow/.local/lib/python3.8/site-packages/docker/api/container.py", line 441, in create_container_from_config return self._result(res, True) File "/home/airflow/.local/lib/python3.8/site-packages/docker/api/client.py", line 274, in _result self._raise_for_status(response) File "/home/airflow/.local/lib/python3.8/site-packages/docker/api/client.py", line 270, in _raise_for_status raise create_api_error_from_http_exception(e) File "/home/airflow/.local/lib/python3.8/site-packages/docker/errors.py", line 31, in create_api_error_from_http_exception raise cls(e, response=response, explanation=explanation) docker.errors.APIError: 400 Client Error for http+docker://localhost/v1.30/containers/create?name=docker_command_sleep: Bad Request ("invalid mount config for type "bind": bind source path does not exist: /tmp/airflowtmpw5gvv6dj") [2021-07-04 23:49:54,222] {taskinstance.py:1544} INFO - Marking task as FAILED. dag_id=docker_eod_us_equities, task_id=docker_command_sleep, execution_date=20210704T231948, start_date=20210704T234953, end_date=20210704T234954 [2021-07-04 23:49:54,297] {local_task_job.py:151} INFO - Task exited with return code 1 ``` I have even tried with Docker API v1.41 (latest) and same issue. I have bound the `/var/run/docker.sock` as a bind mount into the container. **Docker Compose:** ```yaml # Licensed to the Apache Software Foundation (ASF) under one # or more contributor license agreements. See the NOTICE file # distributed with this work for additional information # regarding copyright ownership. The ASF licenses this file # to you under the Apache License, Version 2.0 (the # "License"); you may not use this file except in compliance # with the License. You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, # software distributed under the License is distributed on an # "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY # KIND, either express or implied. See the License for the # specific language governing permissions and limitations # under the License. # # Basic Airflow cluster configuration for CeleryExecutor with Redis and PostgreSQL. # # WARNING: This configuration is for local development. Do not use it in a production deployment. # # This configuration supports basic configuration using environment variables or an .env file # The following variables are supported: # # AIRFLOW_IMAGE_NAME - Docker image name used to run Airflow. # Default: apache/airflow:master-python3.8 # AIRFLOW_UID - User ID in Airflow containers # Default: 50000 # AIRFLOW_GID - Group ID in Airflow containers # Default: 50000 # # Those configurations are useful mostly in case of standalone testing/running Airflow in test/try-out mode # # _AIRFLOW_WWW_USER_USERNAME - Username for the administrator account (if requested). # Default: airflow # _AIRFLOW_WWW_USER_PASSWORD - Password for the administrator account (if requested). # Default: airflow # _PIP_ADDITIONAL_REQUIREMENTS - Additional PIP requirements to add when starting all containers. # Default: '' # # Feel free to modify this file to suit your needs. --- version: '3' x-airflow-common: &airflow-common image: ${AIRFLOW_IMAGE_NAME:-apache/airflow:2.1.1-python3.8} environment: &airflow-common-env AIRFLOW__CORE__EXECUTOR: CeleryExecutor AIRFLOW__CORE__SQL_ALCHEMY_CONN: postgresql+psycopg2://airflow:airflow@postgres/airflow AIRFLOW__CELERY__RESULT_BACKEND: db+postgresql://airflow:airflow@postgres/airflow AIRFLOW__CELERY__BROKER_URL: redis://:@redis:6379/0 AIRFLOW__CORE__FERNET_KEY: '' AIRFLOW__CORE__DAGS_ARE_PAUSED_AT_CREATION: 'true' AIRFLOW__CORE__LOAD_EXAMPLES: 'false' AIRFLOW__SCHEDULER__DAG_DIR_LIST_INTERVAL: 300 # Just to have a fast load in the front-end. Do not use it in production with those configurations. AIRFLOW__API__AUTH_BACKEND: 'airflow.api.auth.backend.basic_auth' AIRFLOW__CORE__ENABLE_XCOM_PICKLING: 'true' # "_run_image of the DockerOperator returns now a python string, not a byte string" Ref: https://github.com/apache/airflow/issues/13487 _PIP_ADDITIONAL_REQUIREMENTS: ${_PIP_ADDITIONAL_REQUIREMENTS:-} volumes: - ./dags:/opt/airflow/dags - ./logs:/opt/airflow/logs - ./plugins:/opt/airflow/plugins - '/var/run/docker.sock:/var/run/docker.sock' # We will pass the Docker Deamon as a volume to allow the webserver containers start docker images. Ref: https://stackoverflow.com/q/51342810/7024760 user: '${AIRFLOW_UID:-50000}:${AIRFLOW_GID:-50000}' depends_on: redis: condition: service_healthy postgres: condition: service_healthy services: postgres: image: postgres:13 environment: POSTGRES_USER: airflow POSTGRES_PASSWORD: airflow POSTGRES_DB: airflow volumes: - postgres-db-volume:/var/lib/postgresql/data healthcheck: test: ['CMD', 'pg_isready', '-U', 'airflow'] interval: 5s retries: 5 restart: always redis: image: redis:latest ports: - 6379:6379 healthcheck: test: ['CMD', 'redis-cli', 'ping'] interval: 5s timeout: 30s retries: 50 restart: always airflow-webserver: <<: *airflow-common command: webserver ports: - 80:8080 healthcheck: test: ['CMD', 'curl', '--fail', 'http://localhost:80/health'] interval: 10s timeout: 10s retries: 5 restart: always airflow-scheduler: <<: *airflow-common command: scheduler healthcheck: test: [ 'CMD-SHELL', 'airflow jobs check --job-type SchedulerJob --hostname "$${HOSTNAME}"', ] interval: 10s timeout: 10s retries: 5 restart: always airflow-worker: <<: *airflow-common command: celery worker healthcheck: test: - 'CMD-SHELL' - 'celery --app airflow.executors.celery_executor.app inspect ping -d "celery@$${HOSTNAME}"' interval: 10s timeout: 10s retries: 5 restart: always airflow-init: <<: *airflow-common command: version environment: <<: *airflow-common-env _AIRFLOW_DB_UPGRADE: 'true' _AIRFLOW_WWW_USER_CREATE: 'true' _AIRFLOW_WWW_USER_USERNAME: ${_AIRFLOW_WWW_USER_USERNAME:-airflow} _AIRFLOW_WWW_USER_PASSWORD: ${_AIRFLOW_WWW_USER_PASSWORD:-airflow} flower: <<: *airflow-common command: celery flower ports: - 5555:5555 healthcheck: test: ['CMD', 'curl', '--fail', 'http://localhost:5555/'] interval: 10s timeout: 10s retries: 5 restart: always volumes: postgres-db-volume: ``` **DAG:** ```python from datetime import datetime, timedelta import pendulum from airflow import DAG from airflow.operators.bash_operator import BashOperator from airflow.providers.docker.operators.docker import DockerOperator from airflow.operators.dummy_operator import DummyOperator AMERICA_NEW_YORK_TIMEZONE = pendulum.timezone('US/Eastern') default_args = { 'owner': 'airflow', 'description': 'Docker testing', 'depend_on_past': False, 'start_date': datetime(2021, 5, 1, tzinfo=AMERICA_NEW_YORK_TIMEZONE), 'retries': 1, 'retry_delay': timedelta(minutes=30), } with DAG( 'docker_test', default_args=default_args, schedule_interval="15 20 * * *", catchup=False, ) as dag: start_dag = DummyOperator(task_id='start_dag') end_dag = DummyOperator(task_id='end_dag') t1 = BashOperator(task_id='print_current_date', bash_command='date') t2 = DockerOperator( task_id='docker_command_sleep', image='alpine', container_name='docker_command_sleep', api_version='1.30', auto_remove=True, command="/bin/sleep 3", docker_url="unix://var/run/docker.sock", network_mode="bridge", do_xcom_push=True, ) start_dag >> t1 t1 >> t2 t2 >> end_dag ``` **Anything else we need to know**: Problem happens anything `DockerOperator` is being used. Not entirely sure why this happening given that the docker sock is fully permissive (has `777`) and is bind mounted into the container. When I test via *docker-py* client in Python shell under `airflow` user inside the container, it works perfectly fine to run all docker-py operations like listing running containers and such confirming the mounted docker UNIX socket is available and working. However, even with the `docker_url` pointing to the docker socket in the above DAG, I am getting this error thrown in above trace. <!-- How often does this problem occur? Once? Every time etc? Any relevant logs to include? Put them here in side a detail tag: <details><summary>x.log</summary> lots of stuff </details> --> -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
