Hey Daniel, > This is not a veto threat, but just expressing concern, I remain a bit skeptical of multitenancy for a few reasons.
Just to bring it back to the right track - this is not a multitenancy, really. It's multi-team. We've been going over that topic and I started to like the "team" approach - and we definitely are not going to run multiple "tenants" just multiple "teams" - in the same organization - where typically today those teams share the same instance of Airflow and have common "pipeline dependencies" between them, while they have different expectations for dependencies (both for parsing and execution) and want to be able to manage the execution resources (not scheduling) separately. > 1. a lot of complexity will be introduced. Somewhat yes. If it's a lot compared to all the other changes we are implementing - I don't think so. The concept evolved "to stand on the shoulders" of all the other changes for Airflow - and I'd say that if they can be done, this one is mostly a "description on how to do team-deployment as part of organization deployment" using existing components and "enabling the concept by introducing shared team_id" that can bind together dag parsing and execution and UI. 2. it's already not terribly uncommon that airflow users find scheduler performance / concurrency insufficient and this would tend to make that issue worse. If someone will try to use it as "multi-tenancy", yes - but again - for a long time this is not a goal (and I understood that tenancy means more for a lot of people, that's why I adopted the "team" and I am going to explain it over and over). This is not going to make it "easy" to manage and add new tenants. Quite the opposite - this should be deliberate effort where organization's manager delegates part of their responsibility (setting up the environment and managing execution resources) to team deployment managers. There will be no "one-click-add-new-tenant" feature built in. it will require reconfiguring your airflow to add a new team. 3. usefulness would seem to be somewhat limited by the requirement that users be on same version of airflow. Again - yes, if you want to run a multi-tenant Airflow. But if you have several teams to manage in your organization, you want to keep the "core airflow" upgraded while giving certain freedom to your teams to manage their environment. J. On Tue, Jul 30, 2024 at 6:04 PM Daniel Standish <daniel.stand...@astronomer.io.invalid> wrote: > This is not a veto threat, but just expressing concern, I remain a bit > skeptical of multitenancy for a few reasons. > > 1. a lot of complexity will be introduced. > 2. it's already not terribly uncommon that airflow users find scheduler > performance / concurrency insufficient and this would tend to make that > issue worse. > 3. usefulness would seem to be somewhat limited by the requirement that > users be on same version of airflow. >
