Hijacking the thread further here, any thoughts on how to breakdown per DAG
access?
Tao & I are talking about introducing per-DAG permissions and one big
question is whether we'll need to support different operation-types at a
per-DAG level, which changes the way we need to model the perms.
First [simpler] option is to introduce one perm per DAG. If you have access
to 5 DAGs, and you have `can_clear` and `can_run`, you'll have homogenous
rights on the DAGs you have access to.
Second option is to have a breakdown per DAG. Meaning for each DAG we
create a set of perms ({dag_id}_can_view, {dag_id}_can_modify, ...). So one
user could have modify on some DAGs, view on others, and other DAGs would
be invisible. This could be broken down further ({dag_id}_can_clear, ...)
but it gets hard to manage.
Thoughts?
Max
On Wed, Mar 28, 2018 at 10:02 PM, Tao Feng <fengta...@gmail.com> wrote:
> Great work Joy. This is awesome! I am interested in helping out the per dag
> level access. Just created a ticket to check(AIRFLOW-2267). Let me know if
> you have any suggestions. I will share my proposal once I am ready.
>
> On Fri, Mar 23, 2018 at 6:45 PM, Joy Gao <j...@wepay.com> wrote:
>
> > Hey guys!
> >
> > The RBAC UI <https://github.com/apache/incubator-airflow/pull/3015> has
> > been merged to master. I'm looking forward to early adopters' feedback
> and
> > bug reports. I also hope to have more folks helping out with the RBAC UI,
> > especially with introducing DAG-Level access control, which is a feature
> > that a lot of people have been asking. If you are interested in helping
> out
> > with this effort, let's talk more!
> >
> > This commit will be in the 1.10.0 release, and we are going to maintain
> > both UIs simultaneously for a short period of time. Once RBAC UI is
> stable
> > and battle-tested, we will deprecate the old UI and eventually remove it
> > from the repo (around Airflow 2.0.0 or 2.1.0 release). This is to prevent
> > two UIs from forking into separate paths, as that would become very
> > difficult to maintain.
> >
> > Going forward while both UIs are up, if you are making a change to any
> > files in airflow/www/ (old UI), where applicable, please also make the
> > change to the airflow/www_rbac/ (new UI). If you rather not make changes
> in
> > both UIs, it is recommended that you only make the changes to the RBAC
> UI,
> > since that is the one we are maintaining in the long term.
> >
> > I'm excited that the RBAC UI will be able to bring additional security to
> > Airflow, and with FAB framework in place we can look into leveraging it
> for
> > a unified set of APIs used by both UI and CLI.
> >
> > Joy
> >
> >
> >
> > On Thu, Feb 8, 2018 at 11:31 AM, Joy Gao <j...@wepay.com> wrote:
> >
> > > Hi folks,
> > >
> > > I have a PR <https://github.com/apache/incubator-airflow/pull/3015>
> out
> > > for the new UI. I've included instructions on how to test it out in the
> > PR
> > > description. Looking forward to your feedbacks.
> > >
> > > Cheers,
> > > Joy
> > >
> > > On Fri, Dec 1, 2017 at 6:18 PM, Joy Gao <j...@wepay.com> wrote:
> > >
> > >> Thanks for the background info. Would be really awesome for you to
> have
> > >> PyPi access :D I'll make the change to have Airflow Webserver's FAB
> > >> dependency pointing to my fork for the mean time.
> > >>
> > >> For folks who are interested in RBAC, I will be giving a talk/demo at
> > the Airflow
> > >> Meet-Up
> > >> <https://www.meetup.com/Bay-Area-Apache-Airflow-Incubating-
> > Meetup/events/244525050/>
> > >> next Monday. Happy to chat afterwards about it as well :)
> > >>
> > >> On Thu, Nov 30, 2017 at 8:36 AM, Maxime Beauchemin <
> > >> maximebeauche...@gmail.com> wrote:
> > >>
> > >>> A bit of related history here:
> > >>> https://github.com/dpgaspar/Flask-AppBuilder/issues/399
> > >>>
> > >>> On Thu, Nov 30, 2017 at 8:33 AM, Maxime Beauchemin <
> > >>> maximebeauche...@gmail.com> wrote:
> > >>>
> > >>> > Given I have merge rights on FAB I could probably do another round
> of
> > >>> > review and get your PRs through. I would really like to get the
> main
> > >>> > maintainer's input on things that touch the core (composite-key
> > >>> support) as
> > >>> > he might have concerns/intuitions that we can't know about.
> > >>> >
> > >>> > I do not have Pypi access though so I cannot push new releases
> out. I
> > >>> > could ask for that.
> > >>> >
> > >>> > I've threatened to fork the project before, that's always an
> option.
> > >>> I've
> > >>> > noticed his involvement is sporadic and comes in bursts.
> > >>> >
> > >>> > In the meantime, you can have the dependency in Airflow Webserver
> > >>> pointing
> > >>> > straight to your fork.
> > >>> >
> > >>> > Max
> > >>> >
> > >>> > On Wed, Nov 29, 2017 at 7:02 PM, Joy Gao <j...@wepay.com> wrote:
> > >>> >
> > >>> >> I just created a new webserver instance if you haven't gotten a
> > >>> chance to
> > >>> >> fiddle around with the new web UI and the RBAC configurations
> > (thanks
> > >>> >> Maxime for getting started with this earlier!):
> > >>> >>
> > >>> >> http://104.209.38.171:8080/
> > >>> >>
> > >>> >> Admin Account
> > >>> >> username: admin
> > >>> >> password: admin
> > >>> >>
> > >>> >> Read-Only Account
> > >>> >> username: viewer
> > >>> >> password: password
> > >>> >>
> > >>> >>
> > >>> >> On Wed, Nov 29, 2017 at 2:58 PM, Joy Gao <j...@wepay.com> wrote:
> > >>> >>
> > >>> >> > Hi folks,
> > >>> >> >
> > >>> >> > Thanks for all the feedback regarding to the new Airflow
> Webserver
> > >>> UI
> > >>> >> > <https://github.com/wepay/airflow-webserver/>! I've been
> actively
> > >>> >> > addressing all the bugs that were raised on Github. So I want to
> > >>> take
> > >>> >> this
> > >>> >> > opportunity to discuss two issues coming up:
> > >>> >> >
> > >>> >> > The first issue is unaddressed PRs in FAB. If these PRs continue
> > to
> > >>> stay
> > >>> >> > unaddressed, RBAC is blocked from making further progress. If
> this
> > >>> >> continue
> > >>> >> > to be an issue, I'm inclined to fork FAB, even though it's not
> > >>> >> idealistic.
> > >>> >> >
> > >>> >> >
> > >>> >> > - PR/631 <https://github.com/dpgaspar/F
> > lask-AppBuilder/pull/631>
> > >>> >> Binary
> > >>> >> > column support (merged, unreleased)
> > >>> >> > <https://github.com/dpgaspar/Flask-AppBuilder/pull/631>
> > >>> >> > - PR/639 <https://github.com/dpgaspar/F
> > lask-AppBuilder/pull/639>
> > >>> >> Composite
> > >>> >> > primary key support (unmerged)
> > >>> >> > - PR/655 <https://github.com/dpgaspar/F
> > lask-AppBuilder/pull/655>
> > >>> >> Form
> > >>> >> > prefill support (unmerged)
> > >>> >> >
> > >>> >> >
> > >>> >> > The second issue is an open question about the next step of
> > Airflow
> > >>> >> > Webserver itself. Here are the 3 potential directions we could
> > >>> take, and
> > >>> >> > I've added my thought on each.
> > >>> >> >
> > >>> >> > 1. Permanently keep Airflow Webserver as a separated package
> from
> > >>> >> Airflow,
> > >>> >> > and treat it as another UI option. Keep `www` in Airflow. Allow
> > >>> >> development
> > >>> >> > on both UIs.
> > >>> >> > *I'm not a fan of this. When there is an existing UI in Airflow,
> > >>> most
> > >>> >> > contributors would prefer to maintain the official version that
> is
> > >>> >> > installed out-of-the-box. **Having a second UI outside of
> Airflow
> > >>> will
> > >>> >> > make maintaining it very difficult, leading to an eventual death
> > of
> > >>> the
> > >>> >> new
> > >>> >> > UI :(*
> > >>> >> >
> > >>> >> > 2. Permanently keep Airflow Webserver as a separated package
> from
> > >>> >> Airflow,
> > >>> >> > but freeze all development on `www` and direct all future UI
> > >>> >> development
> > >>> >> > to Airflow Webserver, eventually removing `www` completely when
> > >>> Airflow
> > >>> >> > Webserver is stable.
> > >>> >> > *I'm not a fan of this either. First of all, the views and
> models
> > >>> are
> > >>> >> > tightly coupled in both old and new UI; until we have a
> > full-fledged
> > >>> >> REST
> > >>> >> > API to build the UI (and cli) on top of it, separating them to a
> > >>> >> separate
> > >>> >> > package now will potentially cause dependency issues and add
> > >>> >> complication
> > >>> >> > to our release cycle. **Secondly, **majority of Airflow users
> run
> > >>> >> Airflow
> > >>> >> > with the UI; it's one of Airflow's best features. Separating UI
> > out
> > >>> of
> > >>> >> > Airflow core will complicate setup and configuration, while
> making
> > >>> >> Airflow
> > >>> >> > core less complete.*
> > >>> >> >
> > >>> >> > 3. Merge Airflow Webserver back into Airflow as `www2`, freeze
> all
> > >>> >> > development on `www`, eventually removing `www` completely when
> > >>> `www2`
> > >>> >> is
> > >>> >> > stable.
> > >>> >> > *This makes the most sense to me. Airflow Webserver is developed
> > >>> with
> > >>> >> the
> > >>> >> > goal of feature parity to the current UI, plus additional RBAC
> > >>> >> capability,
> > >>> >> > in hope to replace the old UI completely. Yes, this means there
> > >>> will be
> > >>> >> a
> > >>> >> > short period of having to maintain two UIs, but once we freeze
> > >>> >> development
> > >>> >> > on www, it shouldn't be a concern for long.*
> > >>> >> >
> > >>> >> > I'd love to hear everyone's thoughts on this! I'm excited about
> > >>> bringing
> > >>> >> > RBAC to airflow and I hope it's something others will find
> useful
> > as
> > >>> >> well!
> > >>> >> >
> > >>> >> > Cheers,
> > >>> >> > Joy
> > >>> >> >
> > >>> >> > On Mon, Nov 20, 2017 at 11:24 AM, Joy Gao <j...@wepay.com>
> wrote:
> > >>> >> >
> > >>> >> >> Thank you everyone for the active feedback so far, and thanks
> for
> > >>> >> setting
> > >>> >> >> up the demo Maxime!
> > >>> >> >>
> > >>> >> >> Going to work on pruning through the issues in the upcoming
> days.
> > >>> >> >>
> > >>> >> >> Fokko/Maxime, do you recall the SQLAlchemy Exception message
> so I
> > >>> can
> > >>> >> >> look into it? Otherwise I'll wait until it's down again =P
> > >>> >> >>
> > >>> >> >> Cheers,
> > >>> >> >>
> > >>> >> >> Joy
> > >>> >> >>
> > >>> >> >> On Mon, Nov 20, 2017 at 9:35 AM, Maxime Beauchemin <
> > >>> >> >> maximebeauche...@gmail.com> wrote:
> > >>> >> >>
> > >>> >> >>> I just restarted it, not sure how long it will take to get in
> a
> > >>> bad
> > >>> >> state
> > >>> >> >>> again...
> > >>> >> >>>
> > >>> >> >>> Max
> > >>> >> >>>
> > >>> >> >>> On Sun, Nov 19, 2017 at 11:55 PM, Driesprong, Fokko
> > >>> >> <fo...@driesprong.frl
> > >>> >> >>> >
> > >>> >> >>> wrote:
> > >>> >> >>>
> > >>> >> >>> > Good morning,
> > >>> >> >>> >
> > >>> >> >>> > The demo provided by Max is down, it throws a
> > >>> SQLAlchemyexception
> > >>> >> :'(
> > >>> >> >>> >
> > >>> >> >>> > Cheers, Fokko
> > >>> >> >>> >
> > >>> >> >>> > 2017-11-18 19:14 GMT+01:00 Chris Riccomini <
> > >>> criccom...@apache.org>:
> > >>> >> >>> >
> > >>> >> >>> > > @bolke, open issues on the Github repo, please.
> > >>> >> >>> > >
> > >>> >> >>> > > On Sat, Nov 18, 2017 at 10:13 AM, Bolke de Bruin <
> > >>> >> bdbr...@gmail.com>
> > >>> >> >>> > > wrote:
> > >>> >> >>> > >
> > >>> >> >>> > > > Chris,
> > >>> >> >>> > > >
> > >>> >> >>> > > > Do you want us to report bugs somewhere (I have
> > encountered
> > >>> a
> > >>> >> >>> few)? Or
> > >>> >> >>> > > > just generic user experiences posted here?
> > >>> >> >>> > > >
> > >>> >> >>> > > > Cheers
> > >>> >> >>> > > > Bolke
> > >>> >> >>> > > >
> > >>> >> >>> > > > > On 18 Nov 2017, at 00:47, Chris Riccomini <
> > >>> >> criccom...@apache.org
> > >>> >> >>> >
> > >>> >> >>> > > wrote:
> > >>> >> >>> > > > >
> > >>> >> >>> > > > > Hey all,
> > >>> >> >>> > > > >
> > >>> >> >>> > > > > I know the weekend is coming up, and for those of us
> in
> > >>> the
> > >>> >> US,
> > >>> >> >>> next
> > >>> >> >>> > > week
> > >>> >> >>> > > > > is a bit of a slow holiday week. Would love to get
> some
> > >>> >> feedback
> > >>> >> >>> from
> > >>> >> >>> > > > > everyone on this. The goal would ideally to be to
> > >>> converge on
> > >>> >> >>> this
> > >>> >> >>> > and
> > >>> >> >>> > > > > eventually replace the existing Airflow UI with this
> > one.
> > >>> >> >>> > > > >
> > >>> >> >>> > > > > Cheers,
> > >>> >> >>> > > > > Chris
> > >>> >> >>> > > > >
> > >>> >> >>> > > > > On Fri, Nov 17, 2017 at 1:44 PM, Joy Gao <
> > j...@wepay.com>
> > >>> >> wrote:
> > >>> >> >>> > > > >
> > >>> >> >>> > > > >> Hi guys.
> > >>> >> >>> > > > >>
> > >>> >> >>> > > > >> I've been working on moving airflow from Flask-Admin
> to
> > >>> >> >>> > > Flask-AppBuilder
> > >>> >> >>> > > > >> for RBAC
> > >>> >> >>> > > > >> <https://cwiki.apache.org/
> confluence/display/AIRFLOW/
> > >>> >> >>> > > > Airflow+RBAC+proposal
> > >>> >> >>> > > > >>> ,
> > >>> >> >>> > > > >> check it out at https://github.com/wepay/airfl
> > >>> ow-webserver.
> > >>> >> >>> > > > >>
> > >>> >> >>> > > > >> It's still a work-in-progress, but most features you
> > see
> > >>> in
> > >>> >> the
> > >>> >> >>> > > > webserver
> > >>> >> >>> > > > >> UI today is available there. For those who are
> > >>> interested in
> > >>> >> >>> RBAC,
> > >>> >> >>> > I'd
> > >>> >> >>> > > > love
> > >>> >> >>> > > > >> to get some early feedback in terms of the following:
> > >>> >> >>> > > > >>
> > >>> >> >>> > > > >> - New Flask-AppBuilder UI (any bugs/regressions)
> > >>> >> >>> > > > >> - Setup issues
> > >>> >> >>> > > > >> - Ease of integration with third party auth (i.e.
> LDAP,
> > >>> AD,
> > >>> >> >>> OAuth,
> > >>> >> >>> > > etc.)
> > >>> >> >>> > > > >> - Any other thoughts/concerns
> > >>> >> >>> > > > >>
> > >>> >> >>> > > > >> Thanks a lot!
> > >>> >> >>> > > > >>
> > >>> >> >>> > > > >> Cheers,
> > >>> >> >>> > > > >> Joy
> > >>> >> >>> > > > >>
> > >>> >> >>> > > >
> > >>> >> >>> > > >
> > >>> >> >>> > >
> > >>> >> >>> >
> > >>> >> >>>
> > >>> >> >>
> > >>> >> >>
> > >>> >> >
> > >>> >>
> > >>> >
> > >>> >
> > >>>
> > >>
> > >>
> > >
> >
>
