Works for me! Dan said he might confcall in. Alex? Max
On Mon, Jul 24, 2017 at 11:25 AM, Chris Riccomini <[email protected]> wrote: > Wednesday 8/2 is perfect. Want to do it like 3-5? I booked a room for 12 > people (and video conferencing) at WePay in this time slot. Should allow > you to head home easily afterwards. :) That work for you guys? > > On Mon, Jul 24, 2017 at 11:01 AM, Maxime Beauchemin < > [email protected]> wrote: > > > The week of the 31st sound good. Wednesday? > > > > About React we may not need a frontend lib like it (or at least not just > > yet). We can talk about it at the meeting. > > > > Max > > > > On Fri, Jul 21, 2017 at 12:47 AM, Bolke de Bruin <[email protected]> > > wrote: > > > > > We avoid React for the same reasons as the ASF and use Polymer 2 > instead. > > > Would that work? > > > > > > Bolke. > > > > > > > On 20 Jul 2017, at 19:35, Chris Riccomini <[email protected]> > > wrote: > > > > > > > > Hey Max, > > > > > > > > Want to come down to WePay? We can set up a zoom for those that want > to > > > > join online, and record it as well to post for the community. > > > > > > > > Since Joy is just getting started, and it looks like there's going to > > be > > > a > > > > K8s discussion next week, maybe we can shoot for the week after (the > > week > > > > of the 31st of July)? Care to float a few times that week? > > > > > > > > Cheers, > > > > Chris > > > > > > > > On Thu, Jul 20, 2017 at 9:31 AM, Maxime Beauchemin < > > > > [email protected]> wrote: > > > > > > > >> Sounds awesome, count me in! > > > >> > > > >> * check out the prototype in my fork, I went far enough to hit some > > > >> hurdles, try different workarounds. I hooked up the Airflow > Bootstrap > > > >> template too so that we feel at home in this new UI > > > >> * using a single `id` field is a requirement for FAB that airflow > > > doesn't > > > >> respect (composite pks), either we add the feature to support that > in > > > FAB, > > > >> or we align on the Airflow side and modify the models and add a > > > migration > > > >> script. This upgrade would require downtime and might be annoying to > > the > > > >> Airflow community, but could help with db performance a bit (smaller > > > >> index)... I probably could be convinced either way but I'm leaning > on > > > >> improving FAB > > > >> * I'm a maintainer for FAB so I can help get stuff through there > > > >> * React is in limbo at the ASF for licensing reasons, so no React at > > > least > > > >> for now > > > >> * npm/webpack/ES6, javascript only in `.js` files > > > >> * I vote for eslint + eslint-config-airbnb as a set of linting rules > > > for JS > > > >> * Keep out of apache (for now), this new app ships as its own pypi > > > package > > > >> `airflow-webserver`, have a period of overlap (maintaining 2 web > apps) > > > >> before ripping out `airflow/www` from the core package > > > >> * You need to get in touch with Marty Kausas, an intern at Airbnb > > who's > > > >> been working on a Flask blueprint for improved, more personalized > > views > > > on > > > >> DAGs that we were planning on merging into the main branch > eventually. > > > Some > > > >> of Marty's idea and code could be merged into this effort. > > > >> > > > >> These are ideas on how I would proceed personally on this but > > definitely > > > >> everything here is up for discussion. > > > >> > > > >> Let's meet physically at either WePay or Airbnb. Folks from the > > > community, > > > >> let us know on this thread if you want to be part of this effort, > > we'll > > > be > > > >> happy to include you. > > > >> > > > >> Thanks, > > > >> > > > >> Max > > > >> > > > >> On Wed, Jul 19, 2017 at 7:33 PM, Joy Gao <[email protected]> wrote: > > > >> > > > >>> Hey everyone, > > > >>> > > > >>> I recently transferred to Data Infra team here at WePay to focus on > > > >>> Airflow-related initiatives. > > > >>> > > > >>> Given the RBAC design is mostly hashed out, I'm happy to get this > > > feature > > > >>> off the ground for Q3, starting with converting Airflow to Fab, if > > > there > > > >>> are no objections. > > > >>> > > > >>> Cheers, > > > >>> Joy > > > >>> > > > >>> On Thu, Jun 29, 2017 at 7:32 AM, Gurer Kiratli < > > > >>> [email protected]> wrote: > > > >>> > > > >>>> Hey all, > > > >>>> > > > >>>> We talked about this internally. We would like to work on this > > feature > > > >>> but > > > >>>> given the immediate priorities we are not going to be working on > it > > in > > > >>> Q3. > > > >>>> Comes end of Q3 we will reevaluate. Likely scenario is we can work > > on > > > >> it > > > >>>> late Q4 or Q12018. > > > >>>> > > > >>>> Cheers, > > > >>>> > > > >>>> Gurer > > > >>>> > > > >>>> On Tue, Jun 27, 2017 at 8:08 AM, Chris Riccomini < > > > >> [email protected]> > > > >>>> wrote: > > > >>>> > > > >>>>> I think FAB sounds like the right approach. Waiting to hear back > > with > > > >>>> notes > > > >>>>> on AirBNB H2 discussion to see if they want to take this up. > > > >>>>> > > > >>>>> @Gurer, any idea when this will happen? > > > >>>>> > > > >>>>> On Thu, Jun 22, 2017 at 1:00 AM, Bolke de Bruin < > [email protected] > > > > > > >>>> wrote: > > > >>>>> > > > >>>>>> One downside I see from FAB is that is does not do Business Role > > > >>>> mapping > > > >>>>>> to FAB role. I would prefer to create groups in IPA/LDAP/AD and > > > >> have > > > >>>>> those > > > >>>>>> map to FAB roles instead of needing to manage that in FAB. > > > >>>>>> > > > >>>>>> B. > > > >>>>>> > > > >>>>>>> On 22 Jun 2017, at 09:36, Bolke de Bruin <[email protected]> > > > >>> wrote: > > > >>>>>>> > > > >>>>>>> Hi Guys, > > > >>>>>>> > > > >>>>>>> Thanks for putting the thinking in! It is about time that we > get > > > >>> this > > > >>>>>> moving. > > > >>>>>>> > > > >>>>>>> The design looks pretty sound. One can argue about the > different > > > >>>> roles > > > >>>>>> that are required, but that will be situation dependent I guess. > > > >>>>>>> > > > >>>>>>> Implementation wise I would argue together with Max that FAB > is a > > > >>>>> better > > > >>>>>> or best fit. The ER model that is being described is pretty > much a > > > >>> copy > > > >>>>> of > > > >>>>>> a normal security model. So a reimplementation of that is 1) > > > >>>> significant > > > >>>>>> duplication of effort and 2) bound to have bugs that have been > > > >> solved > > > >>>> in > > > >>>>>> the other framework. Moreover, FAB does have integration out of > > the > > > >>> box > > > >>>>>> with some enterprisey systems like IPA, ActiveDirectory, and > LDAP. > > > >>>>>>> > > > >>>>>>> So while you argue that using FAB would increase the scope of > the > > > >>>>>> proposal significantly, but I think that is not true. Using FAB > > > >> would > > > >>>>> allow > > > >>>>>> you to focus on what kind of out-of-the-box permission sets and > > > >> roles > > > >>>> we > > > >>>>>> would need and maybe address some issues that FAB lacks (maybe > how > > > >> to > > > >>>>> deal > > > >>>>>> with non web access - ie. in DAGs, maybe Kerberos, probably how > to > > > >>> deal > > > >>>>>> with API calls that are not CRUD). Implementation wise it > probably > > > >>>>>> simplifies what we need to do. Maybe - using Max’s early POC as > an > > > >>>>> example > > > >>>>>> - we can slowly move over? > > > >>>>>>> > > > >>>>>>> On a side note: Im planning to hire 2-3 ppl to work on Airflow > > > >>> coming > > > >>>>>> year. Improvement of Security, Enterprise Integration, Revamp UI > > > >> are > > > >>> on > > > >>>>> the > > > >>>>>> todo list. However, this is not confirmed yet as business > > > >> priorities > > > >>>>> might > > > >>>>>> change. > > > >>>>>>> > > > >>>>>>> Bolke. > > > >>>>>>> > > > >>>>>>> > > > >>>>>>>> On 15 Jun 2017, at 21:45, kalpesh dharwadkar < > > > >>>>>> [email protected]> wrote: > > > >>>>>>>> > > > >>>>>>>> @Dan: > > > >>>>>>>> > > > >>>>>>>> Thanks for your feedback. I will remove the REFRESH_DAG > > > >>> permission. > > > >>>>>>>> > > > >>>>>>>> @Max: > > > >>>>>>>> > > > >>>>>>>> Thanks for your response. > > > >>>>>>>> > > > >>>>>>>> The scope of my proposal was just to add RBAC security feature > > > >> to > > > >>>>>> Airflow > > > >>>>>>>> without replacing any existing frameworks. > > > >>>>>>>> > > > >>>>>>>> I understand that adopting FAB would serve Airflow better > moving > > > >>>>>> forward, > > > >>>>>>>> however porting Airflow to using FAB significantly increases > the > > > >>>> scope > > > >>>>>> of > > > >>>>>>>> the proposal and I don't have the time and expertise to carry > > > >> out > > > >>>> the > > > >>>>>> tasks > > > >>>>>>>> in the extended scope. > > > >>>>>>>> > > > >>>>>>>> Hence, I'm curious to know if there's a plan for Airflow to > > > >>> migrate > > > >>>> to > > > >>>>>> FAB > > > >>>>>>>> this year? > > > >>>>>>>> > > > >>>>>>>> - Kalpesh > > > >>>>>>>> > > > >>>>>>>> On Mon, Jun 12, 2017 at 6:16 PM, Maxime Beauchemin < > > > >>>>>>>> [email protected]> wrote: > > > >>>>>>>> > > > >>>>>>>>> It would be nice to go with a framework for this. I did some > > > >>>>>>>>> experimentation using FlaskAppBuilder to go in this > direction. > > > >> It > > > >>>>>> provides > > > >>>>>>>>> auth on different authentication backends out of the box > > > >> (oauth, > > > >>>>>> openid, > > > >>>>>>>>> ldap, registration, ...), generates perms for each view that > > > >> has > > > >>> an > > > >>>>>>>>> @has_access decorator, generates at set of perms for each ORM > > > >>> model > > > >>>>>> (show, > > > >>>>>>>>> edit, delete, add, ...) and enforces it in the CRUD views as > > > >> well > > > >>>> as > > > >>>>>> in the > > > >>>>>>>>> generated REST api that you get for free as a byprdoduct of > > > >>>> deriving > > > >>>>>> FAB's > > > >>>>>>>>> models (essentially it's SqlAlchemy with a layer on top). > > > >>>>>>>>> > > > >>>>>>>>> I started a POC on FAB here a while ago: > > > >>>>>>>>> https://github.com/mistercrunch/airflow_webserver at the > time > > > >> my > > > >>>>> main > > > >>>>>>>>> motivation was the free/instantaneous REST api. > > > >>>>>>>>> > > > >>>>>>>>> I think FAB is a decent fit as the porting should be fairly > > > >>>>>> straightforward > > > >>>>>>>>> (moving the flask views over and deprecating Flask-Admin in > > > >> favor > > > >>>> of > > > >>>>>> FAB's > > > >>>>>>>>> crud) though there was a few blockers. From memory I think > FAB > > > >>>> didn't > > > >>>>>> like > > > >>>>>>>>> the compound PKs we use in some of the Airflow models. We'd > > > >> have > > > >>> to > > > >>>>>> either > > > >>>>>>>>> write a db migration script on the Airflow side, or add > support > > > >>> for > > > >>>>>>>>> compound keys to FAB (I recently became a maintainer of the > > > >>>> project, > > > >>>>>> so I > > > >>>>>>>>> could help with that) > > > >>>>>>>>> > > > >>>>>>>>> The only downside of FAB is that it's not as mature as > > > >> something > > > >>>> like > > > >>>>>>>>> Django, but porting to Django would surely be much more work. > > > >>>>>>>>> > > > >>>>>>>>> Then there's the flask-security suite, but that looks like a > > > >> bit > > > >>>> of a > > > >>>>>>>>> patchwork to me, I guess we can pick and choose which we want > > > >> to > > > >>>> use. > > > >>>>>>>>> > > > >>>>>>>>> Max > > > >>>>>>>>> > > > >>>>>>>>> On Mon, Jun 12, 2017 at 12:50 PM, Dan Davydov < > > > >>>>>>>>> [email protected]> wrote: > > > >>>>>>>>> > > > >>>>>>>>>> Looks good to me in general, thanks for putting this > together! > > > >>>>>>>>>> > > > >>>>>>>>>> I think the ability to integrate with external RBAC systems > > > >> like > > > >>>>> LDAP > > > >>>>>> is > > > >>>>>>>>>> important (i.e. the Airflow DB should not be decoupled with > > > >> the > > > >>>> RBAC > > > >>>>>>>>>> database wherever possible). > > > >>>>>>>>>> > > > >>>>>>>>>> I wouldn't be too worried about the permissions about > > > >> refreshing > > > >>>>>> DAGs, as > > > >>>>>>>>>> far as I know this functionality is no longer required with > > > >> the > > > >>>> new > > > >>>>>>>>>> webservers which reload state periodically, and will > certainly > > > >>> be > > > >>>>>> removed > > > >>>>>>>>>> when we have a better DAG consistency story. > > > >>>>>>>>>> > > > >>>>>>>>>> I think it would also be good to think about this > > > >>>>>> proposal/implementation > > > >>>>>>>>>> and how it applied in the API-driven world (e.g. when > > > >> webserver > > > >>>> hits > > > >>>>>> APIs > > > >>>>>>>>>> like /clear on behalf of users instead of running commands > > > >>> against > > > >>>>> the > > > >>>>>>>>>> database directly). > > > >>>>>>>>>> > > > >>>>>>>>>> On Mon, Jun 12, 2017 at 11:12 AM, Bolke de Bruin < > > > >>>> [email protected] > > > >>>>>> > > > >>>>>>>>>> wrote: > > > >>>>>>>>>> > > > >>>>>>>>>>> Will respond but im traveling at the moment. Give me a few > > > >>> days. > > > >>>>>>>>>>> > > > >>>>>>>>>>> Sent from my iPhone > > > >>>>>>>>>>> > > > >>>>>>>>>>>> On 12 Jun 2017, at 13:39, Chris Riccomini < > > > >>>> [email protected]> > > > >>>>>>>>>> wrote: > > > >>>>>>>>>>>> > > > >>>>>>>>>>>> Hey all, > > > >>>>>>>>>>>> > > > >>>>>>>>>>>> Checking in on this. We spent a good chunk of time > thinking > > > >>>> about > > > >>>>>>>>> this, > > > >>>>>>>>>>> and > > > >>>>>>>>>>>> want to move forward with it, but want to make sure we're > > > >> all > > > >>> on > > > >>>>> the > > > >>>>>>>>>> same > > > >>>>>>>>>>>> page. > > > >>>>>>>>>>>> > > > >>>>>>>>>>>> Max? Bolke? Dan? Jeremiah? > > > >>>>>>>>>>>> > > > >>>>>>>>>>>> Cheers, > > > >>>>>>>>>>>> Chris > > > >>>>>>>>>>>> > > > >>>>>>>>>>>> On Thu, Jun 8, 2017 at 1:49 PM, kalpesh dharwadkar < > > > >>>>>>>>>>>> [email protected]> wrote: > > > >>>>>>>>>>>> > > > >>>>>>>>>>>>> Hello everyone, > > > >>>>>>>>>>>>> > > > >>>>>>>>>>>>> As you all know, currently Airflow doesn’t have a > built-in > > > >>> Role > > > >>>>>>>>> Based > > > >>>>>>>>>>>>> Access Control(RBAC) capability. It does provide very > > > >>> limited > > > >>>>>>>>>>>>> authorization capability by providing admin, > data_profiler, > > > >>> and > > > >>>>>> user > > > >>>>>>>>>>> roles. > > > >>>>>>>>>>>>> However, associating these roles to authenticated > > > >> identities > > > >>> is > > > >>>>> not > > > >>>>>>>>> a > > > >>>>>>>>>>>>> simple effort. > > > >>>>>>>>>>>>> > > > >>>>>>>>>>>>> To address this issue, I have created a design proposal > for > > > >>>>>> building > > > >>>>>>>>>>> RBAC > > > >>>>>>>>>>>>> into Airflow and simplifying user access management via > the > > > >>>>> Airflow > > > >>>>>>>>>> UI. > > > >>>>>>>>>>>>> > > > >>>>>>>>>>>>> The design proposal is located at > > > >> https://cwiki.apache.org/ > > > >>>>>>>>>>>>> confluence/display/AIRFLOW/Airflow+RBAC+proposal > > > >>>>>>>>>>>>> > > > >>>>>>>>>>>>> Any comments/questions/feedback are much appreciated. > > > >>>>>>>>>>>>> > > > >>>>>>>>>>>>> Thanks > > > >>>>>>>>>>>>> Kalpesh > > > >>>>>>>>>>>>> > > > >>>>>>>>>>> > > > >>>>>>>>>> > > > >>>>>>>>> > > > >>>>>>> > > > >>>>>> > > > >>>>>> > > > >>>>> > > > >>>> > > > >>> > > > >>> > > > >>> > > > >>> -- > > > >>> > > > >>> Joy Gao > > > >>> Software Engineer > > > >>> 350 Convention Way, Suite 200 > > > >>> Redwood City, CA 94063 > > > >>> Mobile: 669-224-9305 > > > >>> > > > >>> Payments partner to the platform economy > > > >>> > > > >> > > > > > > > > >
