litesh1998 opened a new issue, #60621: URL: https://github.com/apache/airflow/issues/60621
### Description I propose creating a new community provider, airflow-provider-uv, to natively integrate the [uv](https://github.com/astral-sh/uv) package manager with Airflow. The core component is the UvOperator, which executes arbitrary commands inside a locked uv environment. It is designed to be completely decoupled from the Airflow worker's environment while offering significant performance advantages over existing virtualenv solutions. The proposed provider would include: - UvOperator: Runs commands (scripts, generic CLI tools, test suites) in a uv project environment. - UvToolOperator: Executes ephemeral tools (e.g., uv tool run dbt-core) without installing them in the project. - UvSyncOperator: Explicitly pre-warms environments to prevent race conditions in high-concurrency DAGs. Key Technical Features: Agnostic Execution: Takes a generic command list, making no assumptions about the code being run. Real-time Log Streaming: Uses selectors to stream stdout and stderr independently to Airflow logs, preserving native "Red" error highlighting in the UI. Native Signal Handling: Implements on_kill to ensure subprocess groups are terminated correctly when tasks are marked as failed. ### Use case/motivation 1. The "Decoupling" Gap Currently, DAG authors often tightly couple business logic to Airflow (using PythonOperator) or take on the heavy operational burden of Docker for every task. - PythonOperator is fast but unsafe (dependency hell). - DockerOperator is isolated but heavy (image build/push/pull overhead). - PythonVirtualenvOperator is safe but slow (recreates venv on every run). 2. The Need for Speed & Agnosticism With the rise of uv, we can achieve "Docker-like" isolation with "Process-like" startup speeds using the global cache. This operator allows users to run any tool—whether it's a Python script, a dbt command, or a pytest suite—without Airflow needing to know the implementation details. ### Related issues _No response_ ### Are you willing to submit a PR? - [x] Yes I am willing to submit a PR! ### Code of Conduct - [x] I agree to follow this project's [Code of Conduct](https://github.com/apache/airflow/blob/main/CODE_OF_CONDUCT.md) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
