GitHub user pykenny edited a comment on the discussion: Separate Python virtual environments for each DAG(not Task)
I read the [slides about remote worker](https://airflowsummit.org/sessions/2024/running-airflow-tasks-anywhere-in-any-language/) (or edge worker, to be more specifically) introduced on last year's Airflow Summit and feel that it could possibly be the solution when the feature becomes fully available. An executor still need to run in an environment where the same Python+Airflow version as the core service does, but at runner level it won't matter as long as it supports the recognized task execution protocol (using compatible Task SDK to communicate with executor). Finally DAG authors can write task flow in this way to specify the environment(s): ```python @dag(...): def my_dag: # ... task_venv = task.external("environment_001") upstream = task_venv.upstream_job() task_venv.downstream_job(data=upstream) # ... ``` And on the machine where the edge executor runs, runners will have access to the following task definitions: ```python from airflow.sdk import task # Just Task SDK installed on the environment, Core not included @task(task_id="upstream_job", **context) def upstream_job() -> str: # ... @task(task_id="downstream_job", **context) def downstream_job(data): # ... ``` Unfortunately at the time of writing, Airflow officials haven't release any SDK that supports language outside of Python (Golang, JavaScript, TypeScript, etc.) yet, and current Python Task SDK has to take care of [backward compatibility of existing codebase](https://airflow.apache.org/blog/airflow-three-point-oh-is-here/#:~:text=The%20Airflow%203%20release%20includes%20the%20Python%20TaskSDK%20which%20enables%20backward%20compatibility%20for%20existing%20DAGs.%20TaskSDKs%20for%20additional%20languages%2C%20starting%20with%20Golang%20will%20be%20released%20over%20the%20next%20few%20months.), meaning it's [heavily tied to the core system itself](https://github.com/apache/airflow/blob/35186bccf3da10d5d01b744966be8156721be5c7/task-sdk/pyproject.toml#L49) (runner using Python SDK should run in an environment with Airflow installed), which doesn't offer the same flexibility as what's demonstrated in the summit. GitHub link: https://github.com/apache/airflow/discussions/40651#discussioncomment-13072576 ---- This is an automatically sent email for commits@airflow.apache.org. To unsubscribe, please send an email to: commits-unsubscr...@airflow.apache.org
