ferruzzi commented on code in PR #62102: URL: https://github.com/apache/airflow/pull/62102#discussion_r2824781937
########## airflow-core/docs/core-concepts/multi-team.rst: ########## @@ -0,0 +1,363 @@ +.. Licensed to the Apache Software Foundation (ASF) under one + or more contributor license agreements. See the NOTICE file + distributed with this work for additional information + regarding copyright ownership. The ASF licenses this file + to you under the Apache License, Version 2.0 (the + "License"); you may not use this file except in compliance + with the License. You may obtain a copy of the License at + + .. http://www.apache.org/licenses/LICENSE-2.0 + + .. Unless required by applicable law or agreed to in writing, + software distributed under the License is distributed on an + "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY + KIND, either express or implied. See the License for the + specific language governing permissions and limitations + under the License. + + +Multi-Team +========== + +.. warning:: + Multi-Team is an experimental/incomplete feature currently in preview. The feature will not be + fully complete until Airflow 3.3 and may be subject to changes without warning based on user feedback. + +|experimental| + +Multi-Team Airflow is a feature that enables organizations to run multiple teams within a single Airflow deployment while providing resource isolation and team-based access controls. This feature is designed for medium to large organizations that need to share Airflow infrastructure across multiple teams while maintaining logical separation of resources. + +.. note:: + + Multi-Team Airflow is different from multi-tenancy. It provides isolation within a single deployment but is not designed for complete tenant separation. All teams share the same Airflow infrastructure, scheduler, and metadata database. + +When to Use Multi-Team Mode +--------------------------- + +Multi-Team mode is designed for medium to large organizations that typically have many discrete teams who need access to an Airflow environment. Often there is a dedicated platform or DevOps team to manage the shared Airflow infrastructure, but this is not required. + +**Use Multi-Team mode when:** + +- You have many teams that need to share Airflow infrastructure +- You need resource isolation (Variables, Connections, Secrets, etc) between teams +- You want separate execution environments per team +- You want separate views per team in the Airflow UI +- You want to minimize operational overhead or cost by sharing a single Airflow deployment + +Core Concepts +------------- + +Teams +^^^^^ + +A **Team** is a logical grouping that represents a group of users within your organization. Teams are in part stored in the Airflow metadata database and serve as the basis for resource isolation. + +Teams within the Airflow database have a very simple structure, only containing one field: + +- **name**: A unique identifier for the team (3-50 characters, alphanumeric with hyphens and underscores) + +Teams are associated with Dag bundles through a separate association table, which links team names to Dag bundle names. + +Dag Bundles and Team Ownership +^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ + +Teams are associated with Dags through **Dag Bundles**. A Dag bundle can be owned by at most one team. When a Dag bundle is assigned to a team: + +- All Dags within that bundle belong to that team +- Tasks in those Dags inherit the team association +- The scheduler uses this relationship to determine which executor to use + +.. note:: + + The relationship chain is: **Task → Dag → Dag Bundle → Team** Review Comment: Can we use Workload here? It'll be a Task or a Callback once my PR is merged. ########## airflow-core/docs/core-concepts/multi-team.rst: ########## @@ -0,0 +1,363 @@ +.. Licensed to the Apache Software Foundation (ASF) under one + or more contributor license agreements. See the NOTICE file + distributed with this work for additional information + regarding copyright ownership. The ASF licenses this file + to you under the Apache License, Version 2.0 (the + "License"); you may not use this file except in compliance + with the License. You may obtain a copy of the License at + + .. http://www.apache.org/licenses/LICENSE-2.0 + + .. Unless required by applicable law or agreed to in writing, + software distributed under the License is distributed on an + "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY + KIND, either express or implied. See the License for the + specific language governing permissions and limitations + under the License. + + +Multi-Team +========== + +.. warning:: + Multi-Team is an experimental/incomplete feature currently in preview. The feature will not be + fully complete until Airflow 3.3 and may be subject to changes without warning based on user feedback. + +|experimental| + +Multi-Team Airflow is a feature that enables organizations to run multiple teams within a single Airflow deployment while providing resource isolation and team-based access controls. This feature is designed for medium to large organizations that need to share Airflow infrastructure across multiple teams while maintaining logical separation of resources. + +.. note:: + + Multi-Team Airflow is different from multi-tenancy. It provides isolation within a single deployment but is not designed for complete tenant separation. All teams share the same Airflow infrastructure, scheduler, and metadata database. + +When to Use Multi-Team Mode +--------------------------- + +Multi-Team mode is designed for medium to large organizations that typically have many discrete teams who need access to an Airflow environment. Often there is a dedicated platform or DevOps team to manage the shared Airflow infrastructure, but this is not required. + +**Use Multi-Team mode when:** + +- You have many teams that need to share Airflow infrastructure +- You need resource isolation (Variables, Connections, Secrets, etc) between teams +- You want separate execution environments per team +- You want separate views per team in the Airflow UI +- You want to minimize operational overhead or cost by sharing a single Airflow deployment + +Core Concepts +------------- + +Teams +^^^^^ + +A **Team** is a logical grouping that represents a group of users within your organization. Teams are in part stored in the Airflow metadata database and serve as the basis for resource isolation. + +Teams within the Airflow database have a very simple structure, only containing one field: + +- **name**: A unique identifier for the team (3-50 characters, alphanumeric with hyphens and underscores) + +Teams are associated with Dag bundles through a separate association table, which links team names to Dag bundle names. + +Dag Bundles and Team Ownership +^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ + +Teams are associated with Dags through **Dag Bundles**. A Dag bundle can be owned by at most one team. When a Dag bundle is assigned to a team: + +- All Dags within that bundle belong to that team +- Tasks in those Dags inherit the team association +- The scheduler uses this relationship to determine which executor to use + +.. note:: + + The relationship chain is: **Task → Dag → Dag Bundle → Team** + +Resource Isolation +^^^^^^^^^^^^^^^^^^ + +When Multi-Team mode is enabled, the following resources can be scoped to specific teams: + +- **Variables**: Team members can only access variables owned by their team or global variables +- **Connections**: Team members can only access connections owned by their team or global connections +- **Pools**: Pools can be assigned to teams + +Resources without a team assignment are considered **global** and accessible to all teams. + +Secrets Backends +"""""""""""""""" + +Airflow's secrets backends, including: environment variables, metastore and local filesystem are team-aware. Custom +Secrets Backends are supported on a case by case basis. + +When a task requests a Variable or Connection, the secrets backend will return a team-specific value, if any. The Review Comment: Same here and in other places below, "workload" or "task or callback"? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
