christianbrugger opened a new issue #16806: URL: https://github.com/apache/airflow/issues/16806
docker.errors.APIError: 400 Client Error for http://192.168.221.9:2375/v1.41/containers/create: Bad Request ("invalid mount config for type "bind": bind source path does not exist: /tmp/airflowtmp7naq_r53") **Apache Airflow version**: v2.1.0 **Environment**: - **Cloud provider or hardware configuration**: ec2 t3a.medium - **OS** (e.g. from /etc/os-release): Ubuntu 18.04.5 LTS - **Kernel** (e.g. `uname -a`): 5.4.0-1051-aws - **Install tools**: sudo pip3 install apache-airflow[mysql,ssh,docker,amazon] - **Others**: python 3.6.9 **What happened**: Task fails with error: ```none docker.errors.APIError: 400 Client Error for http://192.168.1.50:2375/v1.41/containers/create: Bad Request ("invalid mount config for type "bind": bind source path does not exist: /tmp/airflowtmp7naq_r53") ``` **How to reproduce it**: Create an separate EC2 instance and forward the docker daemon: ```shell sudo mkdir -p /etc/systemd/system/docker.service.d sudo touch /etc/systemd/system/docker.service.d/options.conf echo -e """ [Service] ExecStart= ExecStart=/usr/bin/dockerd -H unix:// -H tcp://0.0.0.0:2375 """ >> /etc/systemd/system/docker.service.d/options.conf sudo systemctl daemon-reload sudo systemctl restart docker ``` Create dag with DockerOperator ```python DockerOperator( task_id="run_image", docker_url="tcp://192.168.1.50:2375", image="ubuntu:latest", dag=dag, ) ``` Run the DAG. **Anything else we need to know**: To me it looks like the DockerOperator is creating a temporary directory locally and tries to bind it to the container. However as this is a remote container the directory doesn't exist. here is the code part: ```python class DockerOperator(BaseOperator): ... def _run_image(self) -> Optional[str]: """Run a Docker container with the provided image""" self.log.info('Starting docker container from image %s', self.image) with TemporaryDirectory(prefix='airflowtmp', dir=self.host_tmp_dir) as host_tmp_dir: if not self.cli: raise Exception("The 'cli' should be initialized before!") tmp_mount = Mount(self.tmp_dir, host_tmp_dir, "bind") self.container = self.cli.create_container( command=self.format_command(self.command), name=self.container_name, environment={**self.environment, **self._private_environment}, host_config=self.cli.create_host_config( auto_remove=False, mounts=self.mounts + [tmp_mount], network_mode=self.network_mode, shm_size=self.shm_size, dns=self.dns, dns_search=self.dns_search, cpu_shares=int(round(self.cpus * 1024)), mem_limit=self.mem_limit, cap_add=self.cap_add, extra_hosts=self.extra_hosts, privileged=self.privileged, ), image=self.image, user=self.user, entrypoint=self.format_command(self.entrypoint), working_dir=self.working_dir, tty=self.tty, ) ``` I see no way of disabling this behavior without some major patching. How are you guys using remote docker daemons? Is this a use case? Would it be possible to implement something to allow that? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
