o-nikolas commented on code in PR #62102: URL: https://github.com/apache/airflow/pull/62102#discussion_r2824354425
########## airflow-core/docs/core-concepts/multi-team.rst: ########## @@ -0,0 +1,369 @@ +.. Licensed to the Apache Software Foundation (ASF) under one + or more contributor license agreements. See the NOTICE file + distributed with this work for additional information + regarding copyright ownership. The ASF licenses this file + to you under the Apache License, Version 2.0 (the + "License"); you may not use this file except in compliance + with the License. You may obtain a copy of the License at + + .. http://www.apache.org/licenses/LICENSE-2.0 + + .. Unless required by applicable law or agreed to in writing, + software distributed under the License is distributed on an + "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY + KIND, either express or implied. See the License for the + specific language governing permissions and limitations + under the License. + + +Multi-Team +========== + +.. warning:: + Multi-Team is an experimental/incomplete feature currently in preview. The feature will not be + fully complete until Airflow 3.3 and may be subject to changes without warning based on user feedback. + +|experimental| + +Multi-Team Airflow is a feature that enables organizations to run multiple teams within a single Airflow deployment while providing resource isolation and team-based access controls. This feature is designed for medium to large organizations that need to share Airflow infrastructure across multiple teams while maintaining logical separation of resources. + +.. note:: + + Multi-Team Airflow is different from multi-tenancy. It provides isolation within a single deployment but is not designed for complete tenant separation. All teams share the same Airflow infrastructure, scheduler, and metadata database. + +When to Use Multi-Team Mode +--------------------------- + +Multi-Team mode is designed for medium to large organizations that typically have many discrete teams who need access to an Airflow environment. Often there is a dedicated platform or DevOps team to manage the shared Airflow infrastructure, but this is not required. + +**Use Multi-Team mode when:** + +- You have many teams that need to share Airflow infrastructure +- You need resource isolation (Variables, Connections, Secrets, etc) between teams +- You want separate execution environments per team +- You want separate views per team in the Airflow UI +- You want to minimize operational overhead or cost by sharing a single Airflow deployment + +Core Concepts +------------- + +Teams +^^^^^ + +A **Team** is a logical grouping that represents a group of users within your organization. Teams are in part stored in the Airflow metadata database and serve as the basis for resource isolation. + +Teams within the Airflow database have a very simple structure, only containing one field: + +- **name**: A unique identifier for the team (3-50 characters, alphanumeric with hyphens and underscores) + +Teams are associated with Dag bundles through a separate association table, which links team names to Dag bundle names. + +Dag Bundles and Team Ownership +^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ + +Teams are associated with Dags through **Dag Bundles**. A Dag bundle can be owned by at most one team. When a Dag bundle is assigned to a team: + +- All Dags within that bundle belong to that team +- Tasks in those Dags inherit the team association +- The scheduler uses this relationship to determine which executor to use + +.. note:: + + The relationship chain is: **Task → Dag → Dag Bundle → Team** + +Resource Isolation +^^^^^^^^^^^^^^^^^^ + +When Multi-Team mode is enabled, the following resources can be scoped to specific teams: + +- **Variables**: Team members can only access variables owned by their team or global variables +- **Connections**: Team members can only access connections owned by their team or global connections +- **Pools**: Pools can be assigned to teams + +Resources without a team assignment are considered **global** and accessible to all teams. + +Secrets Backends +"""""""""""""""" + +Airflow's secrets backends (environment variables, metastore, local filesystem, and custom backends) are team-aware. +When a task requests a Variable or Connection, the secrets backend will return a team-specific value, if any. The +backend will automatically resolve the correct value based on the requesting task's team. If no team-specific value +exists, the backend falls back to global values + +.. TODO: Diagram showing resource isolation between teams would be helpful here + +Enabling Multi-Team Mode +------------------------ + +To enable Multi-Team mode, set the following configuration in your ``airflow.cfg``: + +.. code-block:: ini + + [core] + multi_team = True + +Or via environment variable: + +.. code-block:: bash + + export AIRFLOW__CORE__MULTI_TEAM=True + +.. warning:: + + Changing this setting on an existing deployment requires careful planning. + +Creating and Managing Teams +--------------------------- + +Teams are managed using the Airflow CLI. The following commands are available: + +Creating a Team +^^^^^^^^^^^^^^^ + +.. code-block:: bash + + airflow teams create <team_name> + +Team names must be 3-50 characters long and contain only alphanumeric characters, hyphens, and underscores. + +Listing Teams +^^^^^^^^^^^^^ + +.. code-block:: bash + + airflow teams list + +This displays all teams in the deployment with their names. + +Deleting a Team +^^^^^^^^^^^^^^^ + +.. code-block:: bash + + airflow teams delete <team_name> + +Or to skip the confirmation prompt: + +.. code-block:: bash + + airflow teams delete <team_name> --yes + +.. warning:: + + A team cannot be deleted if it has associated resources (Dag bundles, Variables, Connections, or Pools). You must remove these associations first. + +Configuring Team Resources +-------------------------- + +Team-scoped Variables +^^^^^^^^^^^^^^^^^^^^^ + +Variables can be associated with teams when created. Tasks belonging to a team can access: + +1. Variables owned by their team +2. Global variables (no team association) + +When a task requests a variable, the system checks for a team-specific variable first, then falls back to a global variable. Review Comment: Will update! -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
