jhgoebbert opened a new issue, #56545: URL: https://github.com/apache/airflow/issues/56545
### Apache Airflow version 3.1.0 ### If "Other Airflow 2/3 version" selected, which one? _No response_ ### What happened? The `@task.kubernetes` decorator fails when used with non-root container images such as the official `apache/airflow` image. After [PR #28942](https://github.com/apache/airflow/pull/28942), the initialization phase for Kubernetes-decorated tasks creates the directory `/airflow/xcom` (see [providers/cncf/kubernetes/decorators/kubernetes.py#L98](https://github.com/apache/airflow/blob/54bd5d8cd9f6f477cc83445737614dec81c4323c/providers/cncf/kubernetes/src/airflow/providers/cncf/kubernetes/decorators/kubernetes.py#L98)). However, in the default Airflow image the container runs as the **`airflow` user**, (see [HERE](https://github.com/apache/airflow/blob/54bd5d8cd9f6f477cc83445737614dec81c4323c/Dockerfile#L2058)) which does not have write permissions to `/`. As a result, the attempt to create `/airflow/xcom` fails with a permissions error, and the Kubernetes task cannot start. ### What you think should happen instead? The Kubernetes-decorated task should run successfully, even when the image runs as a non-root user. ### How to reproduce 1. Use the official Apache Airflow 3 image, e.g.: ```yaml image: apache/airflow:3.1.0 ``` 2. Define a DAG: ``` from airflow.decorators import dag, task from datetime import datetime @dag(start_date=datetime(2024, 1, 1), schedule=None) def test_dag(): @task.kubernetes(image="apache/airflow:3.1.0") def example_task(): print("Hello from Airflow task") example_task() test_dag() ``` 3. Trigger the DAG. ### Operating System Kubernetes (Airflow charts) ### Versions of Apache Airflow Providers Latest ### Deployment Official Apache Airflow Helm Chart ### Deployment details Airflow version: 3.1.0 Executor: @task.kubernetes) Base image: apache/airflow:3.1.0 Kubernetes version: (v1.32) ### Anything else? The /airflow/xcom directory is currently created unconditionally during initialization, assuming root permissions. Possible approaches to resolve this: - Create the directory under a writable path such as /tmp/airflow/xcom, or - Make the XCom path configurable for Kubernetes-decorated tasks, or - Check for existence and create the directory only if it’s writable by the current user. ### Are you willing to submit PR? - [ ] Yes I am willing to submit a PR! ### Code of Conduct - [x] I agree to follow this project's [Code of Conduct](https://github.com/apache/airflow/blob/main/CODE_OF_CONDUCT.md) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
