Jarek - I totally agree. We had a similar conversation in Dec 2022, and Filip K. from Google suggested [1] calling them "workspaces." But I think most of our users and contributors are used to the word "tenant." Trying a new term like "workspaces" just seems to make things more confusing. For example, when I tried using it with a couple of developers at AWS, who were somewhat familiar with Airflow, it immediately prompted questions about its relation to "tenants."
I also liked how you explained it in the AIP response. It's like when Kubernetes talks about "multi-tenant" [2] and they mean it could be for different customers or different teams. What we’re doing with Airflow is for teams, not really different customers. But it's simpler to keep calling it "multi-tenant," just like Kubernetes does, and make sure we explain that we're talking about different teams. Reference: 1. https://docs.google.com/document/d/1n23h26p4_8F5-Cd0JGLPEnF3gumJ5hw3EpwUljz7HcE/edit?disco=AAAAlo3bv6Q 2. https://kubernetes.io/docs/concepts/security/multi-tenancy/ Shubham On 2024-03-15, 2:50 AM, "Jarek Potiuk" <ja...@potiuk.com <mailto:ja...@potiuk.com>> wrote: CAUTION: This email originated from outside of the organization. Do not click links or open attachments unless you can confirm the sender and know the content is safe. AVERTISSEMENT: Ce courrier électronique provient d’un expéditeur externe. Ne cliquez sur aucun lien et n’ouvrez aucune pièce jointe si vous ne pouvez pas confirmer l’identité de l’expéditeur et si vous n’êtes pas certain que le contenu ne présente aucun risque. Thaks Shubham - that's a really nice and succinct description of the vision behind tht 'Multi-tenant deployment' AIP. The case you described is spot-on where I see there is enough value for the organisation that would like to create such multi-tenant deployment of Airflow. And I think one comment fro TP in the AIP that summaries it well - this is not really `Making Airflow multi-tenant'. It's merely enabling airflow components to be put together in a single multi-tenant deployment - consisting with a number of smaller sub-deployments that achieve a nice IMHO way of being able to decompose monolithic Airflow installation Ina a set of loosely cooperating 'sub-deployments' - achieving some (interesting) centralisation property - while giving enough freedom for tenants to manage their part separately In a way it might be a response to the claims that airflow is this old monolithic style application and it's not the new, cool micro-service /serverless world. Which I think trades some problems with a set of other problems mainly around managing all those micro-sevices to cooperate with each other and introduces a lot of performance and service management problems. But with the architecture described here it is a bit connecting best things if both worlds - splitting Airflow into 'midi-services' - where the architecture of your deploynent nicely follows Conway's Law and decomposition is done at the right boundaries - boundaries where Tenant Deployment Managers might claim as 'theirs' while the 'Airflow Platform Team' keeps the whole Airflow deployment under control because there are clear boundaries what the Platform team is responsible for, and where each Tenant has their own 'kingdom'. I think once we get this one approved and some deployments out there in the wild - we will be able to claim that the architecture is actually the next gen after monolithic, micro-sevices and serverless - bit done better, taking into account some real-life user scenarios and Making it relatively easy to adopt such deployment because of Conway's Law limitations. I thought a lot about Conway's Law playing a role in this whole design and I think your Business Case, Shubham, is a perfect reflection of how well the proposes architecture fits in the Business structures of organisation that would like to deploy it. J. czw., 14 mar 2024, 20:21 użytkownik Mehta, Shubham <shu...@amazon.com.inva <mailto:shu...@amazon.com.inva>lid> napisał: > Hi folks, > > Firstly, thanks Jarek for putting together such a thorough and > well-thought-out proposal. > > > > I am very much in support of the multi-tenancy proposal. Having discussed > this with over 30 customers (AWS and non-AWS), there's a clear desire to > shift focus from the complex management of multiple Airflow environments to > enhancing their capabilities, such as enabling data quality checks and > lineage. This proposal is a significant step towards achieving that goal. > > Acknowledging that not every Airflow user has enough time to thoroughly > review the AIP, I have drafted a user scenario that encapsulates what's > possible with the implementation of multi-tenancy support: > > *---- Scenario: Multi-Tenancy in Apache Airflow at [Rocket] ----* > > [Rocket], a leading [mobile gaming platform], has adeptly structured its > cloud operations using Apache Airflow to provide an efficient and secure > multi-tenant environment for orchestrating their complex workflows. This > approach caters to the diverse needs of their three main user groups: the > Data Engineering team, the Data Science team, and the Data Analytics team. > > All teams share basic Airflow components like the Scheduler and Webserver, > providing centralized management with shared cost. Each team has its own > distinct tenant cluster, offering self-sufficiency, flexibility, and > isolation. The Data Engineering team builds ETL/ELT pipelines and produces > user profile, telemetry, and marketing data. The Analytics team works with > marketing data and user information to build comprehensive dashboards. The > Data Science team uses Kubernetes as their execution environment for > heavy-duty machine learning tasks, producing a churn prediction dataset. > > Members of each team can only see and work with their own workflows. > However, Data engineers are granted access to all tenants, enabling them to > assist with DAG troubleshooting and optimization across all teams. Upon > logging in, users are presented with a tenant-specific view, displaying > only the relevant DAGs and artifacts. For those with multi-tenant access, > seamless navigation between different tenant views is available without the > need for re-authentication. > > > This setup lets each team work independently with their own tools and > data, while also getting help from data engineers when needed. It's secure, > efficient, and user-friendly. > > *Image:* https://imgur.com/gallery/uQNqiVc > <https://imgur.com/gallery/uQNqiVc> (highly recommend reviewing > the image to understand the underlying setup) > > *-----------------------------------------------------------------------------------* > > I’d suggest that interested Airflow users review the scenario and share > your support or concerns on this concept in this thread or AIP. For those > interested in diving deeper into the details, the AIP is available here - > https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-67+Multi-tenant+deployment+of+Airflow+components > > <https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-67+Multi-tenant+deployment+of+Airflow+components> > > > Thanks > Shubham > Product Manager - Amazon MWAA > > > > *From: *Jarek Potiuk <ja...@potiuk.com <mailto:ja...@potiuk.com>> > *Reply-To: *"users@airflow.apache.org <mailto:users@airflow.apache.org>" > <users@airflow.apache.org <mailto:users@airflow.apache.org>> > *Date: *Monday, March 11, 2024 at 4:05 PM > *To: *"d...@airflow.apache.org <mailto:d...@airflow.apache.org>" > <d...@airflow.apache.org <mailto:d...@airflow.apache.org>>, " > users@airflow.apache.org <mailto:users@airflow.apache.org>" > <users@airflow.apache.org <mailto:users@airflow.apache.org>> > *Subject: *RE: [EXTERNAL] [COURRIEL EXTERNE] [DISCUSS] DRAFT AIP-67 > Multi-tenant deployment of Airflow components > > > > *CAUTION*: This email originated from outside of the organization. Do not > click links or open attachments unless you can confirm the sender and know > the content is safe. > > > > *AVERTISSEMENT*: Ce courrier électronique provient d’un expéditeur > externe. Ne cliquez sur aucun lien et n’ouvrez aucune pièce jointe si vous > ne pouvez pas confirmer l’identité de l’expéditeur et si vous n’êtes pas > certain que le contenu ne présente aucun risque. > > > > I have iterated and already got a LOT of comments from a LOT of people > (Thanks everyone who spent time on it ). I'd say the document is out of > draft already, it very much describes the idea of multi-tenancy that I hope > we will be voting on some time in the future. > > > > Taking into account that ~ 30% of people in our survey said they want > "mutl-tenancy" - what I am REALLY interested in is to get honest feedback > about the proposal. Manly: > > > > **"Is this the multi-tenancy you were looking for?" * > > > > Or were you looking for different droids (err, tenants) maybe?. > > > > I do not want to exercise my Jedi skills to influence your opinion, that's > why the document is there (and some people say it's nice, readable and > pretty complete) so that you can judge yourself and give feedback. > > > > The document is here: > https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-67+Multi-tenant+deployment+of+Airflow+components > > <https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-67+Multi-tenant+deployment+of+Airflow+components> > > > > > Feel free to comment here, or in the document. I would love to hear more > voices, and have some ideas what to do next to validate the idea, so please > - engage for now - but also expect some follow-ups. > > > > J. > > > > > > On Wed, Mar 6, 2024 at 9:16 AM Jarek Potiuk <ja...@potiuk.com > <mailto:ja...@potiuk.com>> wrote: > > Sooo.. Seems that it's an AIP time :D I've just published a Draft of > AIP-67: > > Multi-tenant deployment of Airflow components > > > https://cwiki.apache.org/confluence/display/AIRFLOW/%5BDRAFT%5D+AIP-67+Multi-tenant+deployment+of+Airflow+components > > <https://cwiki.apache.org/confluence/display/AIRFLOW/%5BDRAFT%5D+AIP-67+Multi-tenant+deployment+of+Airflow+components> > > This AIP is a bit lighter in detail than the others you could see > from Jed , Nikolas and Maciej. This is really a DRAFT / High Level > idea of Multi-Tenancy that could be implemented as the follow-up after > previous steps of Multi-Tenancy implemented (or being implemented) > right now. > > I decided to - rather than describe all the details now - focus on > the concept of Multitenancy that I wanted to propose. Most of all > explaining the concept, comparing it to current ways of achieving some > forms of multi-tenancy and showing benefits and drawbacks of the > solution and connected costs (i.e. what complexity we need to add to > achieve it). > > When thinking about Multi-tenancy, I realized few things: > > * everyone might understand multi-tenancy differently > * some forms of multi-tenancy are achievable even today > * but - most of all - I started to question myself "Is this what we > can do, enough for some, sufficiently numerous groups of users to call > it a useful feature for them". > > So before we get into more details - my aim is to make sure we are all > at the same page on what we CAN do as a multi-tenancy, and eventually > to decide whether we SHOULD do it. > > Have fun. Bring in comments and feedback. > > More about all the currently active AIPs at today's Town Hall > > BTW. Do you think it's a surprise that 5 AIPS were announced just > before the Town Hall? I think not :D > > J. > > --------------------------------------------------------------------- To unsubscribe, e-mail: users-unsubscr...@airflow.apache.org For additional commands, e-mail: users-h...@airflow.apache.org
