rgriffier opened a new issue, #41445: URL: https://github.com/apache/airflow/issues/41445
### Apache Airflow version Other Airflow 2 version (please specify below) ### If "Other Airflow 2 version" selected, which one? 2.10.0rc1 ### What happened? Using a `DockerSwarmOperator`, with `retrieve_output=True` and `retrieve_output_path='/path/to/pickle/file'`, the content of the file located at `/path/to/pickle/file` is not set as XCom prior to deletion of the container. This functionality is described in the `DockerOperator`[documentation](https://airflow.apache.org/docs/apache-airflow-providers-docker/stable/_api/airflow/providers/docker/operators/docker/index.html#airflow.providers.docker.operators.docker.DockerOperator). As `DockerSwarmOperator` inherits from `DockerOperator` (cf. [source](https://airflow.apache.org/docs/apache-airflow-providers-docker/stable/_modules/airflow/providers/docker/operators/docker_swarm.html#DockerSwarmOperator)), I was expecting the `retrieve_output` feature to work using `DockerSwarmOperator`. ### What you think should happen instead? - The same behavior as when using `DockerOperator`, i.e. the contents of the file located at `/path/to/pickle/file` is set as XCom ("return_value" key) when `retrieve_output=True` and `retrieve_output_path='/path/to/pickle/file'`. ### How to reproduce Below is an example of how to reproduce the issue. The `docker_url` argument must be adapted to the deployment context. The DAG is made up of 2 identical tasks, using either a DockerOperator (`write_xcom_docker`) or a DockerSwarmOperator (`write_xcom_docker_swarm`). In these two tasks : - A python script writes a dictionary to a Pickle file (`/tmp/variable.pickle`). - The path to this Pickle file is associated with the `retrieve_output_path argument` in the Docker(Swarm)Operator. ```py from airflow import DAG from airflow.utils.dates import days_ago from datetime import timedelta from airflow.operators.docker_operator import DockerOperator from airflow.contrib.operators.docker_swarm_operator import DockerSwarmOperator params = { 'dag_id': 'test_xcom_docker_docker_swarm', 'catchup': False, 'max_active_runs': 1, 'default_args': { 'owner': 'airflow', 'start_date': days_ago(1), 'retries': 1, 'retry_delay': timedelta(minutes=5) } } with DAG(**params) as dag: write_xcom_docker = DockerOperator( task_id='write_xcom_docker', docker_url="tcp://docker-proxy:2375", image='python:latest', api_version='auto', command="""python -c ' import os import pickle capitals = { "Canada": "Ottawa", "England": "London", "France": "Paris", "Germany": "Berlin", } file_path = "/tmp/variable.pickle" with open(file_path, "wb+") as file: pickle.dump(capitals, file) ' """, retrieve_output=True, retrieve_output_path='/tmp/variable.pickle') write_xcom_docker_swarm = DockerSwarmOperator( task_id='write_xcom_docker_swarm', docker_url="tcp://docker-proxy:2375", image='python:latest', api_version='auto', command="""python -c ' import os import pickle capitals = { "Canada": "Ottawa", "England": "London", "France": "Paris", "Germany": "Berlin", } file_path = "/tmp/variable.pickle" with open(file_path, "wb+") as file: pickle.dump(capitals, file) ' """, retrieve_output=True, retrieve_output_path='/tmp/variable.pickle') write_xcom_docker >> write_xcom_docker_swarm ``` - For the task using a `DockerOperator`, the content of the Pickle file is set as XCom as expected.  - For the task using a `DockerSwarmOperator`, nothing is set as XCom.  ### Operating System Debian GNU/Linux 11 (bullseye) ### Versions of Apache Airflow Providers apache-airflow-providers-docker==3.12.3 ### Deployment Other Docker-based deployment ### Deployment details Airflow was deployed using Docker Swarm, based on Docker version 26.1.4 (build 5650f9b) ### Anything else? The issue has also been tested in version `2.9.2` (using apache-airflow-providers-docker==3.12.0), with the same result. ### Are you willing to submit PR? - [ ] Yes I am willing to submit a PR! ### Code of Conduct - [X] I agree to follow this project's [Code of Conduct](https://github.com/apache/airflow/blob/main/CODE_OF_CONDUCT.md) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
