Thanks Zacharya! I think this is an interesting idea. It seems to be a simple way to add "multi tenancy" to Airflow. However, I'm afraid that having separate venvs for separate tasks (=teams) solves only dependency management problem not managing multiple teams using the same cluster (different teams still can view, delete, update DAGs of other teams and they share the same db). As far as I know there's no "production grade" way to deploy a single Airflow instance that can be shared by teams. Although I know there are ways to achieve that using GKE cluster with multiple instances of Airflow.
Now, focusing on your proposition. If we will decide that we want to support such a feature we will need to think about retention policy for the venvs. Having multiple venvs will increase disk/memory allocation and we should try to remove venvs that are no longer used. In general we will need a mechanism to manage those venvs (define, create, update, delete). That said I'm not convinced if Airflow should take care of its environment/deployment. In my opinion this is a users' task to make sure that their tasks are executed in the right environment. And I think this is easily achievable with multiple workers/queues and docker images that can be build on CI/CD systems. Cheers, Tomek On Fri, Jan 8, 2021 at 5:26 PM Zacharya Haitin <[email protected]> wrote: > > Hi everyone, > > Currently, there is no easy way to manage an Airflow cluster that contains > multiple teams and python based DAGS. > I submitted the following AIP with a suggestion how to make the venv > management part of the Airflow executors lifecycle and would love to get your > feedback. > > My suggestion should be pretty easy to implement and will help the users with > python packages deployments. > > AIP: > https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-37+Virtualenv+management+inside+Airflow > Issue: https://github.com/apache/airflow/issues/13364 > > Please let me know if you have any questions or suggestions on how to improve > this solution, or if you disagree with my approach. > > Thanks, > Zacharya.
