Hey folks, This is a call to action for the multi-team project. We have had many interested parties reach out, but previously we lacked well-defined and well-scoped areas for contributors to tackle. But we now have some items that could use some help, specifically in the executors space.
Context: Each executor needs changes before it can be used in a multi-team setup. The two major requirements are: 1) Ability to read team-based configuration - e.g., two CeleryExecutors may need different brokers, backends, or autoscaling settings, etc for different teams. 2) The executors must be safe to run concurrently within the same scheduler process - e.g. no shared memory/queues, global DB tables, or shared filesystem, etc. This implies different things for different executors; some may need little to no changes some may need many. The LocalExecutor [1], CeleryExecutor [2] and AwsEcsExecutor [3] have already been updated (or are in progress) to support multi-team. All other executors are outstanding. We welcome folks to get involved with updating executors to support multi-team. I have created a tracking ticket here [4] which has sub-tasks per executor. Please add any that I have missed or leave a comment on the issue. Plenty of help and guidance will be provided! Also, in particular, if anyone is familiar with k8s/KubernetesExecutor, I would love for that one to be available for 3.2 but I don't have the time/bandwidth to complete it personally. Thanks for your time, and I look forward to working with you! Cheers, Niko [1] https://github.com/apache/airflow/pull/59021 [2] https://github.com/apache/airflow/pull/60675 [3] https://github.com/apache/airflow/pull/55003 [4] https://github.com/apache/airflow/issues/60912
