Hey Ash (and others who already commented). I hoped, that writing the
proposal down will finally make us all discuss (and eventually converge) on
what "multi-tenancy" means for Airflow and I am glad we have different
opinions - both about naming and scope of the change.
I think those are two things that are worth to discuss a bit separately:
1) "Naming" - Is it enough for "team separation" to use the name
"multi-tenancy" ? I think that one deserves even something like a poll or
eventually voting to see what's a general perception of what "tenancy" and
particularly "multi-tenancy" is. Over the last few weeks I've been talking
to a number of parties (including some offline talks) and it seems that
"multi-tenancy" has completely different co-notation for people - depending
where they come from. For most maintainers it seems what the proposal is
about is "not multi-tenancy", for those who would like to serve different
customers - it's also not "multi-tenancy", but for most of the "platform
teams" I spoke to (those who manage an internal deployment of airflow for a
big organisation with multiple teams/departments - this is "precisely
multi-tenancy".
I am absolutely happy if we use different name eventually, I am not at all
connected to multi-tenancy as a name, and I am happy with for example
"multi-team airflow deployment". I started to like the name actually as it
more explicitly explains what the proposal is (and it still keeps the
multi-"t..." ). And since there are so many people who get confused about
it - I think I would run quick poll during the voting period where I'd add
("and how would you like it to be named" - and I can go either way (if of
course we agree this is a good AIP to be approved in general).
2) Scope. Yes. I hear your point, and yes - it's actually quite possible
already what you propose with a smaller number of changes. But I've heard
from a number of of the users I spoke to - that they are struggling with
two things:
a) having DAG authors from one team to be able to access credentials for
the other team (connections)
b) having dependencies/environment of one team conflicting with
dependencies of the other team
c) having DAG authors from one team to be able to (accidentally or on
purpose) to influence the code of other teams
I think (and just to distill down the "simplification proposal" - what Ash
is explaining is basically, describing a similar setup as AIP-67 DRAFT now
proposes but without `internal-api` component (and let me know Ash if I got
it right).
As I understand the `simplification` proposal without internal-API - it
would make it possible to address a) - by forcing secrets use (not DB) -
and using a combination of deployment "guidelines" explained in the
proposal: per-team separation of queues/executors/triggerer/workers and
providing different configuration per team. Where each team will have a
different set of credentials to access secrets (this is really the only way
we can prevent DAG authors from accessing other team credentials if they
have unbound access to the DB). Similarly, having support for different
executors for different tenants could address b) much more easily than
today - introducing the concept of "per-team" execution environments. But
IMHO there is no way it could address c).
I agree it would satisfy some user's expectations if we do it this way, And
that's quite appealing as indeed - there is quite some work for the
Internal-API component to be ready, and it would already introduce some of
the elements of the complete AIP-67. And it has the benefit that it could
be introduced more granularly - and put much earlier in the hands of our
users, without a more complete redeployment.
Actually when I think of it this way, I am quite happy to take a gradual
approach to AIP-67 to split it into two stages - similarly to what we do
with versioning:
1) Initial AIP where we allow separate "per-team" configuration, where each
team will have their own executors and per-team configuration (including
secret credentials) -> that one will allow a) and b)
2) Full-blown code-separation AIP - including Internal-API component and
true isolation of DAG Authors from one team impacting the others (Which
might happen as a follow-up).
I think that fits very nicely into the existing AIP, adding a separate
"intermediate" stage where the DB remains non-isolated, but the two parts
(connection credentials from secrets and execution environment) will be
entirely usable without the DB isolation.
It will also increase the chance to deliver a usable solution earlier for
those who will find it "Good enough" and even validate whether the "full
code separation" is still as needed in this case.
Ash - does it sound about right? Also I'd love to hear what others think
seeing this context.
J.
On Wed, Mar 27, 2024 at 10:58 AM Ash Berlin-Taylor <a...@apache.org> wrote:
> Thanks very much Jarek and folks for working on this proposal.
>
> I think we can ultimately get Airflow better as a result, but there was
> something that wasn’t sitting quite right with me, and I think I’ve finally
> managed to articulate it in to words. (I left this as a comment on the wiki
> AIP page but wanted to bring it back to the list too for increased
> visibility)
>
> I started off feeling nervous about calling this multi-tenancy as my view
> is that term should be reserved for something that has stronger boundaries
> betwen the tenants. The ambiguity to me is around how strong the boundary
> between tenants is — and what is the mental model users should have around
> how much grief a bad neighbour could cause (intentionally or not).
> This seems like a fairly big change billed as multi-tenancy but to my mind
> it isn't it doesn’t fit that name, and if we are only talking about a
> per-team boundary (crucially: the boundary is between groups/teams within
> one company) then could we instead meet these needs more directly and less
> ambiguously by adding Variable and Connection RBAC (I.e. think along the
> lines of team/dag prefix-X has permissions to these Secrets only?)
>
> That is to my mind closer to what I feel the intent of this would be. Yes,
> that would mean conn names (et al) are a global namespace. I think for some
> users it would be a benefit – for example access to a central DWH would
> only need to be managed (and rotated) once, not once per team and then
> access given to specific teams. It also doesn't lead to overloading of MT
> meaning multiple things to multiple people.
>
> This could build on all the same things already mentioned in this AIP
> (multiple executors per team, multiple dag parsers etc) but is a different
> way of approaching the problem.
>
> This then also maintains the fact that some things are still going to be a
> global namespace, specifically users. There is one list of users/one IdP
> configured and users exist in one (or more) teams.
>
> I think this might also be a smaller change overall, as the only
> namespacing we'd need would be on DAG ids and that could be enforced by
> policy (as in something like ", and everything else could be managed by
> RBAC etc. In fact you are in some ways already talking about adding this
> sort of thing for the allow list on who can trigger a dataset. So lets
> formalize that and expand it a bit.
>
> The other advantage of approaching it this way is that it adds features
> that would be useful for people who are happy with the current "shared"
> infrastructure approach (i.e. happy with multiple teams in one Airflow,
> using existing DAG level rbac etc) but would like the ability to add
> Connection restriction without forcing them to (their mind) complicate
> their deployment story.
>
>
> > On 18 Mar 2024, at 04:42, Amogh Desai <amoghdesai....@gmail.com> wrote:
> >
> > Thanks Jarek and Shubham for the clarifications.
> >
> > Looking forward to this one!
> >
> > Thanks & Regards,
> > Amogh Desai
> >
> > On Sat, Mar 16, 2024 at 10:10 AM Mehta, Shubham
> <shu...@amazon.com.invalid <mailto:shu...@amazon.com.invalid>>
> > wrote:
> >
> >> Jarek - I totally agree. We had a similar conversation in Dec 2022, and
> >> Filip K. from Google suggested [1] calling them "workspaces." But I
> think
> >> most of our users and contributors are used to the word "tenant."
> Trying a
> >> new term like "workspaces" just seems to make things more confusing. For
> >> example, when I tried using it with a couple of developers at AWS, who
> were
> >> somewhat familiar with Airflow, it immediately prompted questions about
> its
> >> relation to "tenants."
> >>
> >> I also liked how you explained it in the AIP response. It's like when
> >> Kubernetes talks about "multi-tenant" [2] and they mean it could be for
> >> different customers or different teams. What we’re doing with Airflow is
> >> for teams, not really different customers. But it's simpler to keep
> calling
> >> it "multi-tenant," just like Kubernetes does, and make sure we explain
> that
> >> we're talking about different teams.
> >>
> >> Reference:
> >> 1.
> >>
> https://docs.google.com/document/d/1n23h26p4_8F5-Cd0JGLPEnF3gumJ5hw3EpwUljz7HcE/edit?disco=AAAAlo3bv6Q
> >> 2. https://kubernetes.io/docs/concepts/security/multi-tenancy/
> >>
> >> Shubham
> >>
> >> On 2024-03-15, 2:50 AM, "Jarek Potiuk" <ja...@potiuk.com <mailto:
> >> ja...@potiuk.com>> wrote:
> >>
> >>
> >> CAUTION: This email originated from outside of the organization. Do not
> >> click links or open attachments unless you can confirm the sender and
> know
> >> the content is safe.
> >>
> >>
> >>
> >>
> >>
> >>
> >> AVERTISSEMENT: Ce courrier électronique provient d’un expéditeur
> externe.
> >> Ne cliquez sur aucun lien et n’ouvrez aucune pièce jointe si vous ne
> pouvez
> >> pas confirmer l’identité de l’expéditeur et si vous n’êtes pas certain
> que
> >> le contenu ne présente aucun risque.
> >>
> >>
> >>
> >>
> >>
> >>
> >> Thaks Shubham - that's a really nice and succinct description of the
> vision
> >> behind tht 'Multi-tenant deployment' AIP. The case you described is
> spot-on
> >> where I see there is enough value for the organisation that would like
> to
> >> create such multi-tenant deployment of Airflow.
> >>
> >>
> >> And I think one comment fro TP in the AIP that summaries it well - this
> is
> >> not really `Making Airflow multi-tenant'. It's merely enabling airflow
> >> components to be put together in a single multi-tenant deployment -
> >> consisting with a number of smaller sub-deployments that achieve a nice
> >> IMHO way of being able to decompose monolithic Airflow installation Ina
> a
> >> set of loosely cooperating 'sub-deployments' - achieving some
> (interesting)
> >> centralisation property - while giving enough freedom for tenants to
> manage
> >> their part separately
> >>
> >>
> >> In a way it might be a response to the claims that airflow is this old
> >> monolithic style application and it's not the new, cool micro-service
> >> /serverless world. Which I think trades some problems with a set of
> other
> >> problems mainly around managing all those micro-sevices to cooperate
> with
> >> each other and introduces a lot of performance and service management
> >> problems.
> >>
> >>
> >> But with the architecture described here it is a bit connecting best
> things
> >> if both worlds - splitting Airflow into 'midi-services' - where the
> >> architecture of your deploynent nicely follows Conway's Law and
> >> decomposition is done at the right boundaries - boundaries where Tenant
> >> Deployment Managers might claim as 'theirs' while the 'Airflow Platform
> >> Team' keeps the whole Airflow deployment under control because there are
> >> clear boundaries what the Platform team is responsible for, and where
> each
> >> Tenant has their own 'kingdom'.
> >>
> >>
> >> I think once we get this one approved and some deployments out there in
> the
> >> wild - we will be able to claim that the architecture is actually the
> next
> >> gen after monolithic, micro-sevices and serverless - bit done better,
> >> taking into account some real-life user scenarios and Making it
> relatively
> >> easy to adopt such deployment because of Conway's Law limitations.
> >>
> >>
> >> I thought a lot about Conway's Law playing a role in this whole design
> and
> >> I think your Business Case, Shubham, is a perfect reflection of how well
> >> the proposes architecture fits in the Business structures of
> organisation
> >> that would like to deploy it.
> >>
> >>
> >> J.
> >>
> >>
> >> czw., 14 mar 2024, 20:21 użytkownik Mehta, Shubham
> >> <shu...@amazon.com.inva <mailto:shu...@amazon.com.inva>lid> napisał:
> >>
> >>
> >>> Hi folks,
> >>>
> >>> Firstly, thanks Jarek for putting together such a thorough and
> >>> well-thought-out proposal.
> >>>
> >>>
> >>>
> >>> I am very much in support of the multi-tenancy proposal. Having
> discussed
> >>> this with over 30 customers (AWS and non-AWS), there's a clear desire
> to
> >>> shift focus from the complex management of multiple Airflow
> environments
> >> to
> >>> enhancing their capabilities, such as enabling data quality checks and
> >>> lineage. This proposal is a significant step towards achieving that
> goal.
> >>>
> >>> Acknowledging that not every Airflow user has enough time to thoroughly
> >>> review the AIP, I have drafted a user scenario that encapsulates what's
> >>> possible with the implementation of multi-tenancy support:
> >>>
> >>> *---- Scenario: Multi-Tenancy in Apache Airflow at [Rocket] ----*
> >>>
> >>> [Rocket], a leading [mobile gaming platform], has adeptly structured
> its
> >>> cloud operations using Apache Airflow to provide an efficient and
> secure
> >>> multi-tenant environment for orchestrating their complex workflows.
> This
> >>> approach caters to the diverse needs of their three main user groups:
> the
> >>> Data Engineering team, the Data Science team, and the Data Analytics
> >> team.
> >>>
> >>> All teams share basic Airflow components like the Scheduler and
> >> Webserver,
> >>> providing centralized management with shared cost. Each team has its
> own
> >>> distinct tenant cluster, offering self-sufficiency, flexibility, and
> >>> isolation. The Data Engineering team builds ETL/ELT pipelines and
> >> produces
> >>> user profile, telemetry, and marketing data. The Analytics team works
> >> with
> >>> marketing data and user information to build comprehensive dashboards.
> >> The
> >>> Data Science team uses Kubernetes as their execution environment for
> >>> heavy-duty machine learning tasks, producing a churn prediction
> dataset.
> >>>
> >>> Members of each team can only see and work with their own workflows.
> >>> However, Data engineers are granted access to all tenants, enabling
> them
> >> to
> >>> assist with DAG troubleshooting and optimization across all teams. Upon
> >>> logging in, users are presented with a tenant-specific view, displaying
> >>> only the relevant DAGs and artifacts. For those with multi-tenant
> access,
> >>> seamless navigation between different tenant views is available without
> >> the
> >>> need for re-authentication.
> >>>
> >>>
> >>> This setup lets each team work independently with their own tools and
> >>> data, while also getting help from data engineers when needed. It's
> >> secure,
> >>> efficient, and user-friendly.
> >>>
> >>> *Image:* https://imgur.com/gallery/uQNqiVc <
> >> https://imgur.com/gallery/uQNqiVc> (highly recommend reviewing
> >>> the image to understand the underlying setup)
> >>>
> >>>
> >>
> *-----------------------------------------------------------------------------------*
> >>>
> >>> I’d suggest that interested Airflow users review the scenario and share
> >>> your support or concerns on this concept in this thread or AIP. For
> those
> >>> interested in diving deeper into the details, the AIP is available
> here -
> >>>
> >>
> https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-67+Multi-tenant+deployment+of+Airflow+components
> >> <
> >>
> https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-67+Multi-tenant+deployment+of+Airflow+components
> >>>
> >>>
> >>>
> >>> Thanks
> >>> Shubham
> >>> Product Manager - Amazon MWAA
> >>>
> >>>
> >>>
> >>> *From: *Jarek Potiuk <ja...@potiuk.com <mailto:ja...@potiuk.com>>
> >>> *Reply-To: *"us...@airflow.apache.org <mailto:us...@airflow.apache.org
> >"
> >> <us...@airflow.apache.org <mailto:us...@airflow.apache.org>>
> >>> *Date: *Monday, March 11, 2024 at 4:05 PM
> >>> *To: *"dev@airflow.apache.org <mailto:dev@airflow.apache.org>" <
> >> dev@airflow.apache.org <mailto:dev@airflow.apache.org>>, "
> >>> us...@airflow.apache.org <mailto:us...@airflow.apache.org>" <
> >> us...@airflow.apache.org <mailto:us...@airflow.apache.org>>
> >>> *Subject: *RE: [EXTERNAL] [COURRIEL EXTERNE] [DISCUSS] DRAFT AIP-67
> >>> Multi-tenant deployment of Airflow components
> >>>
> >>>
> >>>
> >>> *CAUTION*: This email originated from outside of the organization. Do
> not
> >>> click links or open attachments unless you can confirm the sender and
> >> know
> >>> the content is safe.
> >>>
> >>>
> >>>
> >>> *AVERTISSEMENT*: Ce courrier électronique provient d’un expéditeur
> >>> externe. Ne cliquez sur aucun lien et n’ouvrez aucune pièce jointe si
> >> vous
> >>> ne pouvez pas confirmer l’identité de l’expéditeur et si vous n’êtes
> pas
> >>> certain que le contenu ne présente aucun risque.
> >>>
> >>>
> >>>
> >>> I have iterated and already got a LOT of comments from a LOT of people
> >>> (Thanks everyone who spent time on it ). I'd say the document is out of
> >>> draft already, it very much describes the idea of multi-tenancy that I
> >> hope
> >>> we will be voting on some time in the future.
> >>>
> >>>
> >>>
> >>> Taking into account that ~ 30% of people in our survey said they want
> >>> "mutl-tenancy" - what I am REALLY interested in is to get honest
> feedback
> >>> about the proposal. Manly:
> >>>
> >>>
> >>>
> >>> **"Is this the multi-tenancy you were looking for?" *
> >>>
> >>>
> >>>
> >>> Or were you looking for different droids (err, tenants) maybe?.
> >>>
> >>>
> >>>
> >>> I do not want to exercise my Jedi skills to influence your opinion,
> >> that's
> >>> why the document is there (and some people say it's nice, readable and
> >>> pretty complete) so that you can judge yourself and give feedback.
> >>>
> >>>
> >>>
> >>> The document is here:
> >>>
> >>
> https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-67+Multi-tenant+deployment+of+Airflow+components
> >> <
> >>
> https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-67+Multi-tenant+deployment+of+Airflow+components
> >>>
> >>>
> >>>
> >>>
> >>>
> >>> Feel free to comment here, or in the document. I would love to hear
> more
> >>> voices, and have some ideas what to do next to validate the idea, so
> >> please
> >>> - engage for now - but also expect some follow-ups.
> >>>
> >>>
> >>>
> >>> J.
> >>>
> >>>
> >>>
> >>>
> >>>
> >>> On Wed, Mar 6, 2024 at 9:16 AM Jarek Potiuk <ja...@potiuk.com <mailto:
> >> ja...@potiuk.com>> wrote:
> >>>
> >>> Sooo.. Seems that it's an AIP time :D I've just published a Draft of
> >>> AIP-67:
> >>>
> >>> Multi-tenant deployment of Airflow components
> >>>
> >>>
> >>>
> >>
> https://cwiki.apache.org/confluence/display/AIRFLOW/%5BDRAFT%5D+AIP-67+Multi-tenant+deployment+of+Airflow+components
> >> <
> >>
> https://cwiki.apache.org/confluence/display/AIRFLOW/%5BDRAFT%5D+AIP-67+Multi-tenant+deployment+of+Airflow+components
> >>>
> >>>
> >>> This AIP is a bit lighter in detail than the others you could see
> >>> from Jed , Nikolas and Maciej. This is really a DRAFT / High Level
> >>> idea of Multi-Tenancy that could be implemented as the follow-up after
> >>> previous steps of Multi-Tenancy implemented (or being implemented)
> >>> right now.
> >>>
> >>> I decided to - rather than describe all the details now - focus on
> >>> the concept of Multitenancy that I wanted to propose. Most of all
> >>> explaining the concept, comparing it to current ways of achieving some
> >>> forms of multi-tenancy and showing benefits and drawbacks of the
> >>> solution and connected costs (i.e. what complexity we need to add to
> >>> achieve it).
> >>>
> >>> When thinking about Multi-tenancy, I realized few things:
> >>>
> >>> * everyone might understand multi-tenancy differently
> >>> * some forms of multi-tenancy are achievable even today
> >>> * but - most of all - I started to question myself "Is this what we
> >>> can do, enough for some, sufficiently numerous groups of users to call
> >>> it a useful feature for them".
> >>>
> >>> So before we get into more details - my aim is to make sure we are all
> >>> at the same page on what we CAN do as a multi-tenancy, and eventually
> >>> to decide whether we SHOULD do it.
> >>>
> >>> Have fun. Bring in comments and feedback.
> >>>
> >>> More about all the currently active AIPs at today's Town Hall
> >>>
> >>> BTW. Do you think it's a surprise that 5 AIPS were announced just
> >>> before the Town Hall? I think not :D
> >>>
> >>> J.
> >>>
> >>>
> >>
> >>
> >>
> >>
> >> ---------------------------------------------------------------------
> >> To unsubscribe, e-mail: dev-unsubscr...@airflow.apache.org <mailto:
> dev-unsubscr...@airflow.apache.org>
> >> For additional commands, e-mail: dev-h...@airflow.apache.org <mailto:
> dev-h...@airflow.apache.org>
>
