Hey Vikram, Don’t worry about the delay and thanks for sharing your thoughts!
My overall feeling here tends to agree with you (after a discussion with Jarek I confess __). I like this idea of separating the user management to external providers, it allows more features and more user management models to be implemented. Overall I feel this is the right way to go on the long run. I am still not clear on some implementation details but I guess it is too early for that (if we chose this direction). The changes suggested in our proposal would be basically shifted to outside of Airflow so I am trying to convince myself we'll still be able to cherry-pick some of the suggestions we proposed __ (as opposed to entirely lost). However, the changes/idea you are proposing is quite impactful in terms of architecture for Airflow and direction taken so I would really love to hear other feedbacks and opinion on that topic. Vincent On 2023-02-13, 2:04 PM, "Jarek Potiuk" <[email protected]> wrote: Hey Vikram, I think it's brilliant and I wonder how it happened that had not occurred to us earlier. And I believe that is due to the natural tendency of "following as we always did" rather than thinking completely out-of-the-box. Thanks Vikram for bringing it up. The funny thing is that when I see this: > However, I don't agree that this level of user management belongs in "Core Airflow". I almost immediately think - NOOOOO, why, it's always been here, how can we remove it? But then if you look a bit closer: > think this is a time to consider the concept of a "user management provider" with a simple built-in implementation being the current Airflow functionality, enabling alternate more complex (but separate) implementations such as your proposal here as alternate user management providers. Then it starts to make way more sense. Way more. And when you look further: > Maybe, this also enables us to get rid of the Fab security manager from core Airflow? My heart jumps and I am immediately sold on the idea. When I was commenting on the doc initially, something was not right. I had a feeling It is probably the 5th time I am looking and commenting on a similar document. And, well, I did, actually. Most of the things we discussed there are already implemented out there. We just need to make sure we expose enough of the API to use them. For example we have Keycloak that is an open source implementation of Identity and Access Management. With everything out there already integrated. and I've been part of the project that integrated just the authentication part. Now if we rethink the authorization and make it simpler and "externally driven", this will not only be faster IMHO, but also will allow enterprise users to integrate much better. I believe following the path that Vikram outlined will be a good direction for everyone in the community - including all the Manage Service providers, who will have a far easier job on integrating Airflow into their authentication models. J. On Mon, Feb 13, 2023 at 6:24 PM Vikram Koka <[email protected]> wrote: > > Shubham and Vincent, > > Let me start by saying that I apologize for my delayed response to your original email. > > I appreciate the detailed write-up and the thought behind it. I completely agree with your use case and understand how this is applicable to enterprises with multiple data teams using Airflow. > > However, I don't agree that this level of user management belongs in "Core Airflow". > > I strongly believe that the core Airflow mission is for the community at large and for data practitioners either individuals or teams within enterprises. And therefore, I don't disagree with the intent of making it easier for enterprise teams to adopt Airflow. But, I think there is a never ending list of user management features which are needed to support Enterprise needs. We have already struggled with this over time and faced challenges with the Fab security manager and its integration in Airflow. > > I think we should use this opportunity and your use case to "separate the user management" from Core Airflow outside of the absolute basics. I think this is a time to consider the concept of a "user management provider" with a simple built-in implementation being the current Airflow functionality, enabling alternate more complex (but separate) implementations such as your proposal here as alternate user management providers. Maybe, this also enables us to get rid of the Fab security manager from core Airflow? > > Best regards, > Vikram > > > On Fri, Feb 3, 2023 at 8:22 AM Beck, Vincent <[email protected]> wrote: >> >> Thanks __ >> >> On 2023-02-03, 10:55 AM, "Jarek Potiuk" <[email protected]> wrote: >> >> CAUTION: This email originated from outside of the organization. Do not click links or open attachments unless you can confirm the sender and know the content is safe. >> >> >> >> Added. >> >> On Fri, Feb 3, 2023 at 3:53 PM Beck, Vincent >> <[email protected]> wrote: >> > >> > Thank you! https://cwiki.apache.org/confluence/display/~vin100.beck >> > >> > On 2023-02-02, 5:38 PM, "Jarek Potiuk" <[email protected]> wrote: >> > >> > CAUTION: This email originated from outside of the organization. Do not click links or open attachments unless you can confirm the sender and know the content is safe. >> > >> > >> > >> > What's your cwiki ID, Vincent (I'll add you without going into details yet) >> > >>
