I'd suggest something else that avoids having to add a 3rd column. I think we can fit our use case into the existing structure nicely.
My idea is to mimic what FAB does with its own Models. When you create a Model and ModelView in FAB (say DagRun for example), it creates a new view_menu (DagRun) and matches it with existing permission (can_read, can_delete, can_edit, can_show, ...) to create a new set of "permission_views" which are combinations of permission and view_menu as in "DagRun - can_read", "DagRun - can_edit", ... It's not a cartesian product of all perms and view_menus, it's a predetermined list of model-specific perms that get combined with DagRun here. Similarly, when Airflow would detect a new DAG (example "my_dag"), it would create a new view_menu `my_dag`, and match it with permissions that are identified as "combinable" with DAG. To avoid potential conflicts I'd create new permissions that are DAG-specific like "dag_clear", "dag_run", "dag_view". I'm arguing about the how to use the FAB model here specifically. I think this allows for all the flexibility we need without changing the model. I care less about what exactly the atomicity of the per DAG perms should look like. As far as I'm concerned, per-DAG read and write is probably granular enough Also note that: * we need an "_all_dags" logical DAG, meaning we'd have extra permission_views "_all_dags - dag_view", "_all_dags - dag_run", "all_dags - dag_clear" * we probably want to derive FAB's SecurityManger and have an AirflowSecurityManager that has an extra method "can_dag_action(user, dag_id, action)". The SecurityManger class is neat because people can provide their own if they want and override methods. That means that you can defer any of the security-related checks to another system if you want to. Many companies have some sort of company RBAC system and that can be used to integrate with it. Max On Fri, Mar 30, 2018 at 4:54 PM, Joy Gao <[email protected]> wrote: > Hi all, > > I also agree that having view-only access to some dags while write access > to other dags is useful, so I prefer option 2. Although option 2 is more > difficult to manage, it is cleaner and more consistent with the current > security model. (On the other hand, even though option 1 may be may be > easier to manage, it might be trickier to implement: with one perm per > dag, it breaks the existing FAB security model since the existing, more > granular permissions will have to be grouped for each dag). > > What Brian suggested in the other thread makes sense to me... > > The current security model in FAB looks like the following: > > --- permissions > user --- role --- permission_view ---| > --- views > > FAB generates a few relationship tables to manage the mappings, one of > them is *ab_permission_view_role*, which has all the > role-to-permission_view mapping. We can add a dag_id column to this > relationship that expresses dag-level granularity, so the security model > becomes: > > --- > permissions > | > user --- role --- dag_permission_view --- --- views > | > --- dags > > We can make the dag_id field an optional field in the relationship, and > only lazily add new dag-level mappings for DAGs that specify 'access > control'. This way we don't have to create new permissions or views. (We > will still have to introduce new dag-level roles on top of the 5 generic > roles (public/viewer/user/op/admin)). > > (I think this is similar to what Arthur suggested earlier, but not sure if > I interpreted correctly). > > Joy > > On Thu, Mar 29, 2018 at 10:01 AM, Arthur Wiedmer <[email protected] > > wrote: > >> (Creating a new thread) >> >> Hi Max, >> >> I was just wondering about this. There are definite use cases for people >> having only view access to some DAGs, mostly for monitoring. I want to know >> what the upstream DAGs are doing, but maybe I don't need clear/run access. >> >> I feel like the granular operation permissions will start to become >> unwieldy fast. It grows as (# users * # DAGs * # operations) but with >> hopefully a small constant factor in front: most people should only have a >> small number of DAGs they care about. The Ops team has a need to have >> access to all on the other hand. >> >> I was wondering we could get by with something slightly more complex but >> easier on the size of that permissions table : >> 1) One toggle on the user level for broad access (ALL:ALL, RUN/CLEAR:ALL, >> VIEW:ALL) default NULL >> 2) More granular permissions at the DAG level. >> >> So in order, check the user's broad level permission first, then DAG >> level. For large amounts of DAGs, this should help shave a little bit from >> that table. >> >> Best, >> Arthur >> >> >> On Thu, Mar 29, 2018 at 8:27 AM, Maxime Beauchemin < >> [email protected]> wrote: >> >>> Hijacking the thread further here, any thoughts on how to breakdown per >>> DAG >>> access? >>> >>> Tao & I are talking about introducing per-DAG permissions and one big >>> question is whether we'll need to support different operation-types at a >>> per-DAG level, which changes the way we need to model the perms. >>> >>> First [simpler] option is to introduce one perm per DAG. If you have >>> access >>> to 5 DAGs, and you have `can_clear` and `can_run`, you'll have homogenous >>> rights on the DAGs you have access to. >>> >>> Second option is to have a breakdown per DAG. Meaning for each DAG we >>> create a set of perms ({dag_id}_can_view, {dag_id}_can_modify, ...). So >>> one >>> user could have modify on some DAGs, view on others, and other DAGs would >>> be invisible. This could be broken down further ({dag_id}_can_clear, ...) >>> but it gets hard to manage. >>> >>> Thoughts? >>> >>> Max >>> >>> On Wed, Mar 28, 2018 at 10:02 PM, Tao Feng <[email protected]> wrote: >>> >>> > Great work Joy. This is awesome! I am interested in helping out the >>> per dag >>> > level access. Just created a ticket to check(AIRFLOW-2267). Let me >>> know if >>> > you have any suggestions. I will share my proposal once I am ready. >>> > >>> > On Fri, Mar 23, 2018 at 6:45 PM, Joy Gao <[email protected]> wrote: >>> > >>> > > Hey guys! >>> > > >>> > > The RBAC UI <https://github.com/apache/incubator-airflow/pull/3015> >>> has >>> > > been merged to master. I'm looking forward to early adopters' >>> feedback >>> > and >>> > > bug reports. I also hope to have more folks helping out with the >>> RBAC UI, >>> > > especially with introducing DAG-Level access control, which is a >>> feature >>> > > that a lot of people have been asking. If you are interested in >>> helping >>> > out >>> > > with this effort, let's talk more! >>> > > >>> > > This commit will be in the 1.10.0 release, and we are going to >>> maintain >>> > > both UIs simultaneously for a short period of time. Once RBAC UI is >>> > stable >>> > > and battle-tested, we will deprecate the old UI and eventually >>> remove it >>> > > from the repo (around Airflow 2.0.0 or 2.1.0 release). This is to >>> prevent >>> > > two UIs from forking into separate paths, as that would become very >>> > > difficult to maintain. >>> > > >>> > > Going forward while both UIs are up, if you are making a change to >>> any >>> > > files in airflow/www/ (old UI), where applicable, please also make >>> the >>> > > change to the airflow/www_rbac/ (new UI). If you rather not make >>> changes >>> > in >>> > > both UIs, it is recommended that you only make the changes to the >>> RBAC >>> > UI, >>> > > since that is the one we are maintaining in the long term. >>> > > >>> > > I'm excited that the RBAC UI will be able to bring additional >>> security to >>> > > Airflow, and with FAB framework in place we can look into leveraging >>> it >>> > for >>> > > a unified set of APIs used by both UI and CLI. >>> > > >>> > > Joy >>> > > >>> > > >>> > > >>> > > On Thu, Feb 8, 2018 at 11:31 AM, Joy Gao <[email protected]> wrote: >>> > > >>> > > > Hi folks, >>> > > > >>> > > > I have a PR <https://github.com/apache/incubator-airflow/pull/3015 >>> > >>> > out >>> > > > for the new UI. I've included instructions on how to test it out >>> in the >>> > > PR >>> > > > description. Looking forward to your feedbacks. >>> > > > >>> > > > Cheers, >>> > > > Joy >>> > > > >>> > > > On Fri, Dec 1, 2017 at 6:18 PM, Joy Gao <[email protected]> wrote: >>> > > > >>> > > >> Thanks for the background info. Would be really awesome for you to >>> > have >>> > > >> PyPi access :D I'll make the change to have Airflow Webserver's >>> FAB >>> > > >> dependency pointing to my fork for the mean time. >>> > > >> >>> > > >> For folks who are interested in RBAC, I will be giving a >>> talk/demo at >>> > > the Airflow >>> > > >> Meet-Up >>> > > >> <https://www.meetup.com/Bay-Area-Apache-Airflow-Incubating- >>> > > Meetup/events/244525050/> >>> > > >> next Monday. Happy to chat afterwards about it as well :) >>> > > >> >>> > > >> On Thu, Nov 30, 2017 at 8:36 AM, Maxime Beauchemin < >>> > > >> [email protected]> wrote: >>> > > >> >>> > > >>> A bit of related history here: >>> > > >>> https://github.com/dpgaspar/Flask-AppBuilder/issues/399 >>> > > >>> >>> > > >>> On Thu, Nov 30, 2017 at 8:33 AM, Maxime Beauchemin < >>> > > >>> [email protected]> wrote: >>> > > >>> >>> > > >>> > Given I have merge rights on FAB I could probably do another >>> round >>> > of >>> > > >>> > review and get your PRs through. I would really like to get the >>> > main >>> > > >>> > maintainer's input on things that touch the core (composite-key >>> > > >>> support) as >>> > > >>> > he might have concerns/intuitions that we can't know about. >>> > > >>> > >>> > > >>> > I do not have Pypi access though so I cannot push new releases >>> > out. I >>> > > >>> > could ask for that. >>> > > >>> > >>> > > >>> > I've threatened to fork the project before, that's always an >>> > option. >>> > > >>> I've >>> > > >>> > noticed his involvement is sporadic and comes in bursts. >>> > > >>> > >>> > > >>> > In the meantime, you can have the dependency in Airflow >>> Webserver >>> > > >>> pointing >>> > > >>> > straight to your fork. >>> > > >>> > >>> > > >>> > Max >>> > > >>> > >>> > > >>> > On Wed, Nov 29, 2017 at 7:02 PM, Joy Gao <[email protected]> >>> wrote: >>> > > >>> > >>> > > >>> >> I just created a new webserver instance if you haven't gotten >>> a >>> > > >>> chance to >>> > > >>> >> fiddle around with the new web UI and the RBAC configurations >>> > > (thanks >>> > > >>> >> Maxime for getting started with this earlier!): >>> > > >>> >> >>> > > >>> >> http://104.209.38.171:8080/ >>> > > >>> >> >>> > > >>> >> Admin Account >>> > > >>> >> username: admin >>> > > >>> >> password: admin >>> > > >>> >> >>> > > >>> >> Read-Only Account >>> > > >>> >> username: viewer >>> > > >>> >> password: password >>> > > >>> >> >>> > > >>> >> >>> > > >>> >> On Wed, Nov 29, 2017 at 2:58 PM, Joy Gao <[email protected]> >>> wrote: >>> > > >>> >> >>> > > >>> >> > Hi folks, >>> > > >>> >> > >>> > > >>> >> > Thanks for all the feedback regarding to the new Airflow >>> > Webserver >>> > > >>> UI >>> > > >>> >> > <https://github.com/wepay/airflow-webserver/>! I've been >>> > actively >>> > > >>> >> > addressing all the bugs that were raised on Github. So I >>> want to >>> > > >>> take >>> > > >>> >> this >>> > > >>> >> > opportunity to discuss two issues coming up: >>> > > >>> >> > >>> > > >>> >> > The first issue is unaddressed PRs in FAB. If these PRs >>> continue >>> > > to >>> > > >>> stay >>> > > >>> >> > unaddressed, RBAC is blocked from making further progress. >>> If >>> > this >>> > > >>> >> continue >>> > > >>> >> > to be an issue, I'm inclined to fork FAB, even though it's >>> not >>> > > >>> >> idealistic. >>> > > >>> >> > >>> > > >>> >> > >>> > > >>> >> > - PR/631 <https://github.com/dpgaspar/F >>> > > lask-AppBuilder/pull/631> >>> > > >>> >> Binary >>> > > >>> >> > column support (merged, unreleased) >>> > > >>> >> > <https://github.com/dpgaspar/Flask-AppBuilder/pull/631> >>> > > >>> >> > - PR/639 <https://github.com/dpgaspar/F >>> > > lask-AppBuilder/pull/639> >>> > > >>> >> Composite >>> > > >>> >> > primary key support (unmerged) >>> > > >>> >> > - PR/655 <https://github.com/dpgaspar/F >>> > > lask-AppBuilder/pull/655> >>> > > >>> >> Form >>> > > >>> >> > prefill support (unmerged) >>> > > >>> >> > >>> > > >>> >> > >>> > > >>> >> > The second issue is an open question about the next step of >>> > > Airflow >>> > > >>> >> > Webserver itself. Here are the 3 potential directions we >>> could >>> > > >>> take, and >>> > > >>> >> > I've added my thought on each. >>> > > >>> >> > >>> > > >>> >> > 1. Permanently keep Airflow Webserver as a separated package >>> > from >>> > > >>> >> Airflow, >>> > > >>> >> > and treat it as another UI option. Keep `www` in Airflow. >>> Allow >>> > > >>> >> development >>> > > >>> >> > on both UIs. >>> > > >>> >> > *I'm not a fan of this. When there is an existing UI in >>> Airflow, >>> > > >>> most >>> > > >>> >> > contributors would prefer to maintain the official version >>> that >>> > is >>> > > >>> >> > installed out-of-the-box. **Having a second UI outside of >>> > Airflow >>> > > >>> will >>> > > >>> >> > make maintaining it very difficult, leading to an eventual >>> death >>> > > of >>> > > >>> the >>> > > >>> >> new >>> > > >>> >> > UI :(* >>> > > >>> >> > >>> > > >>> >> > 2. Permanently keep Airflow Webserver as a separated package >>> > from >>> > > >>> >> Airflow, >>> > > >>> >> > but freeze all development on `www` and direct all future >>> UI >>> > > >>> >> development >>> > > >>> >> > to Airflow Webserver, eventually removing `www` completely >>> when >>> > > >>> Airflow >>> > > >>> >> > Webserver is stable. >>> > > >>> >> > *I'm not a fan of this either. First of all, the views and >>> > models >>> > > >>> are >>> > > >>> >> > tightly coupled in both old and new UI; until we have a >>> > > full-fledged >>> > > >>> >> REST >>> > > >>> >> > API to build the UI (and cli) on top of it, separating them >>> to a >>> > > >>> >> separate >>> > > >>> >> > package now will potentially cause dependency issues and add >>> > > >>> >> complication >>> > > >>> >> > to our release cycle. **Secondly, **majority of Airflow >>> users >>> > run >>> > > >>> >> Airflow >>> > > >>> >> > with the UI; it's one of Airflow's best features. >>> Separating UI >>> > > out >>> > > >>> of >>> > > >>> >> > Airflow core will complicate setup and configuration, while >>> > making >>> > > >>> >> Airflow >>> > > >>> >> > core less complete.* >>> > > >>> >> > >>> > > >>> >> > 3. Merge Airflow Webserver back into Airflow as `www2`, >>> freeze >>> > all >>> > > >>> >> > development on `www`, eventually removing `www` completely >>> when >>> > > >>> `www2` >>> > > >>> >> is >>> > > >>> >> > stable. >>> > > >>> >> > *This makes the most sense to me. Airflow Webserver is >>> developed >>> > > >>> with >>> > > >>> >> the >>> > > >>> >> > goal of feature parity to the current UI, plus additional >>> RBAC >>> > > >>> >> capability, >>> > > >>> >> > in hope to replace the old UI completely. Yes, this means >>> there >>> > > >>> will be >>> > > >>> >> a >>> > > >>> >> > short period of having to maintain two UIs, but once we >>> freeze >>> > > >>> >> development >>> > > >>> >> > on www, it shouldn't be a concern for long.* >>> > > >>> >> > >>> > > >>> >> > I'd love to hear everyone's thoughts on this! I'm excited >>> about >>> > > >>> bringing >>> > > >>> >> > RBAC to airflow and I hope it's something others will find >>> > useful >>> > > as >>> > > >>> >> well! >>> > > >>> >> > >>> > > >>> >> > Cheers, >>> > > >>> >> > Joy >>> > > >>> >> > >>> > > >>> >> > On Mon, Nov 20, 2017 at 11:24 AM, Joy Gao <[email protected]> >>> > wrote: >>> > > >>> >> > >>> > > >>> >> >> Thank you everyone for the active feedback so far, and >>> thanks >>> > for >>> > > >>> >> setting >>> > > >>> >> >> up the demo Maxime! >>> > > >>> >> >> >>> > > >>> >> >> Going to work on pruning through the issues in the upcoming >>> > days. >>> > > >>> >> >> >>> > > >>> >> >> Fokko/Maxime, do you recall the SQLAlchemy Exception >>> message >>> > so I >>> > > >>> can >>> > > >>> >> >> look into it? Otherwise I'll wait until it's down again =P >>> > > >>> >> >> >>> > > >>> >> >> Cheers, >>> > > >>> >> >> >>> > > >>> >> >> Joy >>> > > >>> >> >> >>> > > >>> >> >> On Mon, Nov 20, 2017 at 9:35 AM, Maxime Beauchemin < >>> > > >>> >> >> [email protected]> wrote: >>> > > >>> >> >> >>> > > >>> >> >>> I just restarted it, not sure how long it will take to >>> get in >>> > a >>> > > >>> bad >>> > > >>> >> state >>> > > >>> >> >>> again... >>> > > >>> >> >>> >>> > > >>> >> >>> Max >>> > > >>> >> >>> >>> > > >>> >> >>> On Sun, Nov 19, 2017 at 11:55 PM, Driesprong, Fokko >>> > > >>> >> <[email protected] >>> > > >>> >> >>> > >>> > > >>> >> >>> wrote: >>> > > >>> >> >>> >>> > > >>> >> >>> > Good morning, >>> > > >>> >> >>> > >>> > > >>> >> >>> > The demo provided by Max is down, it throws a >>> > > >>> SQLAlchemyexception >>> > > >>> >> :'( >>> > > >>> >> >>> > >>> > > >>> >> >>> > Cheers, Fokko >>> > > >>> >> >>> > >>> > > >>> >> >>> > 2017-11-18 19:14 GMT+01:00 Chris Riccomini < >>> > > >>> [email protected]>: >>> > > >>> >> >>> > >>> > > >>> >> >>> > > @bolke, open issues on the Github repo, please. >>> > > >>> >> >>> > > >>> > > >>> >> >>> > > On Sat, Nov 18, 2017 at 10:13 AM, Bolke de Bruin < >>> > > >>> >> [email protected]> >>> > > >>> >> >>> > > wrote: >>> > > >>> >> >>> > > >>> > > >>> >> >>> > > > Chris, >>> > > >>> >> >>> > > > >>> > > >>> >> >>> > > > Do you want us to report bugs somewhere (I have >>> > > encountered >>> > > >>> a >>> > > >>> >> >>> few)? Or >>> > > >>> >> >>> > > > just generic user experiences posted here? >>> > > >>> >> >>> > > > >>> > > >>> >> >>> > > > Cheers >>> > > >>> >> >>> > > > Bolke >>> > > >>> >> >>> > > > >>> > > >>> >> >>> > > > > On 18 Nov 2017, at 00:47, Chris Riccomini < >>> > > >>> >> [email protected] >>> > > >>> >> >>> > >>> > > >>> >> >>> > > wrote: >>> > > >>> >> >>> > > > > >>> > > >>> >> >>> > > > > Hey all, >>> > > >>> >> >>> > > > > >>> > > >>> >> >>> > > > > I know the weekend is coming up, and for those of >>> us >>> > in >>> > > >>> the >>> > > >>> >> US, >>> > > >>> >> >>> next >>> > > >>> >> >>> > > week >>> > > >>> >> >>> > > > > is a bit of a slow holiday week. Would love to get >>> > some >>> > > >>> >> feedback >>> > > >>> >> >>> from >>> > > >>> >> >>> > > > > everyone on this. The goal would ideally to be to >>> > > >>> converge on >>> > > >>> >> >>> this >>> > > >>> >> >>> > and >>> > > >>> >> >>> > > > > eventually replace the existing Airflow UI with >>> this >>> > > one. >>> > > >>> >> >>> > > > > >>> > > >>> >> >>> > > > > Cheers, >>> > > >>> >> >>> > > > > Chris >>> > > >>> >> >>> > > > > >>> > > >>> >> >>> > > > > On Fri, Nov 17, 2017 at 1:44 PM, Joy Gao < >>> > > [email protected]> >>> > > >>> >> wrote: >>> > > >>> >> >>> > > > > >>> > > >>> >> >>> > > > >> Hi guys. >>> > > >>> >> >>> > > > >> >>> > > >>> >> >>> > > > >> I've been working on moving airflow from >>> Flask-Admin >>> > to >>> > > >>> >> >>> > > Flask-AppBuilder >>> > > >>> >> >>> > > > >> for RBAC >>> > > >>> >> >>> > > > >> <https://cwiki.apache.org/ >>> > confluence/display/AIRFLOW/ >>> > > >>> >> >>> > > > Airflow+RBAC+proposal >>> > > >>> >> >>> > > > >>> , >>> > > >>> >> >>> > > > >> check it out at https://github.com/wepay/airfl >>> > > >>> ow-webserver. >>> > > >>> >> >>> > > > >> >>> > > >>> >> >>> > > > >> It's still a work-in-progress, but most features >>> you >>> > > see >>> > > >>> in >>> > > >>> >> the >>> > > >>> >> >>> > > > webserver >>> > > >>> >> >>> > > > >> UI today is available there. For those who are >>> > > >>> interested in >>> > > >>> >> >>> RBAC, >>> > > >>> >> >>> > I'd >>> > > >>> >> >>> > > > love >>> > > >>> >> >>> > > > >> to get some early feedback in terms of the >>> following: >>> > > >>> >> >>> > > > >> >>> > > >>> >> >>> > > > >> - New Flask-AppBuilder UI (any bugs/regressions) >>> > > >>> >> >>> > > > >> - Setup issues >>> > > >>> >> >>> > > > >> - Ease of integration with third party auth (i.e. >>> > LDAP, >>> > > >>> AD, >>> > > >>> >> >>> OAuth, >>> > > >>> >> >>> > > etc.) >>> > > >>> >> >>> > > > >> - Any other thoughts/concerns >>> > > >>> >> >>> > > > >> >>> > > >>> >> >>> > > > >> Thanks a lot! >>> > > >>> >> >>> > > > >> >>> > > >>> >> >>> > > > >> Cheers, >>> > > >>> >> >>> > > > >> Joy >>> > > >>> >> >>> > > > >> >>> > > >>> >> >>> > > > >>> > > >>> >> >>> > > > >>> > > >>> >> >>> > > >>> > > >>> >> >>> > >>> > > >>> >> >>> >>> > > >>> >> >> >>> > > >>> >> >> >>> > > >>> >> > >>> > > >>> >> >>> > > >>> > >>> > > >>> > >>> > > >>> >>> > > >> >>> > > >> >>> > > > >>> > > >>> > >>> >> >> >
