Yeah, I could call in but I probably won't be able to come down that day. On Tue, Jul 25, 2017 at 1:36 PM, Maxime Beauchemin < [email protected]> wrote:
> Works for me! Dan said he might confcall in. Alex? > > Max > > On Mon, Jul 24, 2017 at 11:25 AM, Chris Riccomini <[email protected]> > wrote: > > > Wednesday 8/2 is perfect. Want to do it like 3-5? I booked a room for 12 > > people (and video conferencing) at WePay in this time slot. Should allow > > you to head home easily afterwards. :) That work for you guys? > > > > On Mon, Jul 24, 2017 at 11:01 AM, Maxime Beauchemin < > > [email protected]> wrote: > > > > > The week of the 31st sound good. Wednesday? > > > > > > About React we may not need a frontend lib like it (or at least not > just > > > yet). We can talk about it at the meeting. > > > > > > Max > > > > > > On Fri, Jul 21, 2017 at 12:47 AM, Bolke de Bruin <[email protected]> > > > wrote: > > > > > > > We avoid React for the same reasons as the ASF and use Polymer 2 > > instead. > > > > Would that work? > > > > > > > > Bolke. > > > > > > > > > On 20 Jul 2017, at 19:35, Chris Riccomini <[email protected]> > > > wrote: > > > > > > > > > > Hey Max, > > > > > > > > > > Want to come down to WePay? We can set up a zoom for those that > want > > to > > > > > join online, and record it as well to post for the community. > > > > > > > > > > Since Joy is just getting started, and it looks like there's going > to > > > be > > > > a > > > > > K8s discussion next week, maybe we can shoot for the week after > (the > > > week > > > > > of the 31st of July)? Care to float a few times that week? > > > > > > > > > > Cheers, > > > > > Chris > > > > > > > > > > On Thu, Jul 20, 2017 at 9:31 AM, Maxime Beauchemin < > > > > > [email protected]> wrote: > > > > > > > > > >> Sounds awesome, count me in! > > > > >> > > > > >> * check out the prototype in my fork, I went far enough to hit > some > > > > >> hurdles, try different workarounds. I hooked up the Airflow > > Bootstrap > > > > >> template too so that we feel at home in this new UI > > > > >> * using a single `id` field is a requirement for FAB that airflow > > > > doesn't > > > > >> respect (composite pks), either we add the feature to support that > > in > > > > FAB, > > > > >> or we align on the Airflow side and modify the models and add a > > > > migration > > > > >> script. This upgrade would require downtime and might be annoying > to > > > the > > > > >> Airflow community, but could help with db performance a bit > (smaller > > > > >> index)... I probably could be convinced either way but I'm leaning > > on > > > > >> improving FAB > > > > >> * I'm a maintainer for FAB so I can help get stuff through there > > > > >> * React is in limbo at the ASF for licensing reasons, so no React > at > > > > least > > > > >> for now > > > > >> * npm/webpack/ES6, javascript only in `.js` files > > > > >> * I vote for eslint + eslint-config-airbnb as a set of linting > rules > > > > for JS > > > > >> * Keep out of apache (for now), this new app ships as its own pypi > > > > package > > > > >> `airflow-webserver`, have a period of overlap (maintaining 2 web > > apps) > > > > >> before ripping out `airflow/www` from the core package > > > > >> * You need to get in touch with Marty Kausas, an intern at Airbnb > > > who's > > > > >> been working on a Flask blueprint for improved, more personalized > > > views > > > > on > > > > >> DAGs that we were planning on merging into the main branch > > eventually. > > > > Some > > > > >> of Marty's idea and code could be merged into this effort. > > > > >> > > > > >> These are ideas on how I would proceed personally on this but > > > definitely > > > > >> everything here is up for discussion. > > > > >> > > > > >> Let's meet physically at either WePay or Airbnb. Folks from the > > > > community, > > > > >> let us know on this thread if you want to be part of this effort, > > > we'll > > > > be > > > > >> happy to include you. > > > > >> > > > > >> Thanks, > > > > >> > > > > >> Max > > > > >> > > > > >> On Wed, Jul 19, 2017 at 7:33 PM, Joy Gao <[email protected]> wrote: > > > > >> > > > > >>> Hey everyone, > > > > >>> > > > > >>> I recently transferred to Data Infra team here at WePay to focus > on > > > > >>> Airflow-related initiatives. > > > > >>> > > > > >>> Given the RBAC design is mostly hashed out, I'm happy to get this > > > > feature > > > > >>> off the ground for Q3, starting with converting Airflow to Fab, > if > > > > there > > > > >>> are no objections. > > > > >>> > > > > >>> Cheers, > > > > >>> Joy > > > > >>> > > > > >>> On Thu, Jun 29, 2017 at 7:32 AM, Gurer Kiratli < > > > > >>> [email protected]> wrote: > > > > >>> > > > > >>>> Hey all, > > > > >>>> > > > > >>>> We talked about this internally. We would like to work on this > > > feature > > > > >>> but > > > > >>>> given the immediate priorities we are not going to be working on > > it > > > in > > > > >>> Q3. > > > > >>>> Comes end of Q3 we will reevaluate. Likely scenario is we can > work > > > on > > > > >> it > > > > >>>> late Q4 or Q12018. > > > > >>>> > > > > >>>> Cheers, > > > > >>>> > > > > >>>> Gurer > > > > >>>> > > > > >>>> On Tue, Jun 27, 2017 at 8:08 AM, Chris Riccomini < > > > > >> [email protected]> > > > > >>>> wrote: > > > > >>>> > > > > >>>>> I think FAB sounds like the right approach. Waiting to hear > back > > > with > > > > >>>> notes > > > > >>>>> on AirBNB H2 discussion to see if they want to take this up. > > > > >>>>> > > > > >>>>> @Gurer, any idea when this will happen? > > > > >>>>> > > > > >>>>> On Thu, Jun 22, 2017 at 1:00 AM, Bolke de Bruin < > > [email protected] > > > > > > > > >>>> wrote: > > > > >>>>> > > > > >>>>>> One downside I see from FAB is that is does not do Business > Role > > > > >>>> mapping > > > > >>>>>> to FAB role. I would prefer to create groups in IPA/LDAP/AD > and > > > > >> have > > > > >>>>> those > > > > >>>>>> map to FAB roles instead of needing to manage that in FAB. > > > > >>>>>> > > > > >>>>>> B. > > > > >>>>>> > > > > >>>>>>> On 22 Jun 2017, at 09:36, Bolke de Bruin <[email protected]> > > > > >>> wrote: > > > > >>>>>>> > > > > >>>>>>> Hi Guys, > > > > >>>>>>> > > > > >>>>>>> Thanks for putting the thinking in! It is about time that we > > get > > > > >>> this > > > > >>>>>> moving. > > > > >>>>>>> > > > > >>>>>>> The design looks pretty sound. One can argue about the > > different > > > > >>>> roles > > > > >>>>>> that are required, but that will be situation dependent I > guess. > > > > >>>>>>> > > > > >>>>>>> Implementation wise I would argue together with Max that FAB > > is a > > > > >>>>> better > > > > >>>>>> or best fit. The ER model that is being described is pretty > > much a > > > > >>> copy > > > > >>>>> of > > > > >>>>>> a normal security model. So a reimplementation of that is 1) > > > > >>>> significant > > > > >>>>>> duplication of effort and 2) bound to have bugs that have been > > > > >> solved > > > > >>>> in > > > > >>>>>> the other framework. Moreover, FAB does have integration out > of > > > the > > > > >>> box > > > > >>>>>> with some enterprisey systems like IPA, ActiveDirectory, and > > LDAP. > > > > >>>>>>> > > > > >>>>>>> So while you argue that using FAB would increase the scope of > > the > > > > >>>>>> proposal significantly, but I think that is not true. Using > FAB > > > > >> would > > > > >>>>> allow > > > > >>>>>> you to focus on what kind of out-of-the-box permission sets > and > > > > >> roles > > > > >>>> we > > > > >>>>>> would need and maybe address some issues that FAB lacks (maybe > > how > > > > >> to > > > > >>>>> deal > > > > >>>>>> with non web access - ie. in DAGs, maybe Kerberos, probably > how > > to > > > > >>> deal > > > > >>>>>> with API calls that are not CRUD). Implementation wise it > > probably > > > > >>>>>> simplifies what we need to do. Maybe - using Max’s early POC > as > > an > > > > >>>>> example > > > > >>>>>> - we can slowly move over? > > > > >>>>>>> > > > > >>>>>>> On a side note: Im planning to hire 2-3 ppl to work on > Airflow > > > > >>> coming > > > > >>>>>> year. Improvement of Security, Enterprise Integration, Revamp > UI > > > > >> are > > > > >>> on > > > > >>>>> the > > > > >>>>>> todo list. However, this is not confirmed yet as business > > > > >> priorities > > > > >>>>> might > > > > >>>>>> change. > > > > >>>>>>> > > > > >>>>>>> Bolke. > > > > >>>>>>> > > > > >>>>>>> > > > > >>>>>>>> On 15 Jun 2017, at 21:45, kalpesh dharwadkar < > > > > >>>>>> [email protected]> wrote: > > > > >>>>>>>> > > > > >>>>>>>> @Dan: > > > > >>>>>>>> > > > > >>>>>>>> Thanks for your feedback. I will remove the REFRESH_DAG > > > > >>> permission. > > > > >>>>>>>> > > > > >>>>>>>> @Max: > > > > >>>>>>>> > > > > >>>>>>>> Thanks for your response. > > > > >>>>>>>> > > > > >>>>>>>> The scope of my proposal was just to add RBAC security > feature > > > > >> to > > > > >>>>>> Airflow > > > > >>>>>>>> without replacing any existing frameworks. > > > > >>>>>>>> > > > > >>>>>>>> I understand that adopting FAB would serve Airflow better > > moving > > > > >>>>>> forward, > > > > >>>>>>>> however porting Airflow to using FAB significantly increases > > the > > > > >>>> scope > > > > >>>>>> of > > > > >>>>>>>> the proposal and I don't have the time and expertise to > carry > > > > >> out > > > > >>>> the > > > > >>>>>> tasks > > > > >>>>>>>> in the extended scope. > > > > >>>>>>>> > > > > >>>>>>>> Hence, I'm curious to know if there's a plan for Airflow to > > > > >>> migrate > > > > >>>> to > > > > >>>>>> FAB > > > > >>>>>>>> this year? > > > > >>>>>>>> > > > > >>>>>>>> - Kalpesh > > > > >>>>>>>> > > > > >>>>>>>> On Mon, Jun 12, 2017 at 6:16 PM, Maxime Beauchemin < > > > > >>>>>>>> [email protected]> wrote: > > > > >>>>>>>> > > > > >>>>>>>>> It would be nice to go with a framework for this. I did > some > > > > >>>>>>>>> experimentation using FlaskAppBuilder to go in this > > direction. > > > > >> It > > > > >>>>>> provides > > > > >>>>>>>>> auth on different authentication backends out of the box > > > > >> (oauth, > > > > >>>>>> openid, > > > > >>>>>>>>> ldap, registration, ...), generates perms for each view > that > > > > >> has > > > > >>> an > > > > >>>>>>>>> @has_access decorator, generates at set of perms for each > ORM > > > > >>> model > > > > >>>>>> (show, > > > > >>>>>>>>> edit, delete, add, ...) and enforces it in the CRUD views > as > > > > >> well > > > > >>>> as > > > > >>>>>> in the > > > > >>>>>>>>> generated REST api that you get for free as a byprdoduct of > > > > >>>> deriving > > > > >>>>>> FAB's > > > > >>>>>>>>> models (essentially it's SqlAlchemy with a layer on top). > > > > >>>>>>>>> > > > > >>>>>>>>> I started a POC on FAB here a while ago: > > > > >>>>>>>>> https://github.com/mistercrunch/airflow_webserver at the > > time > > > > >> my > > > > >>>>> main > > > > >>>>>>>>> motivation was the free/instantaneous REST api. > > > > >>>>>>>>> > > > > >>>>>>>>> I think FAB is a decent fit as the porting should be fairly > > > > >>>>>> straightforward > > > > >>>>>>>>> (moving the flask views over and deprecating Flask-Admin in > > > > >> favor > > > > >>>> of > > > > >>>>>> FAB's > > > > >>>>>>>>> crud) though there was a few blockers. From memory I think > > FAB > > > > >>>> didn't > > > > >>>>>> like > > > > >>>>>>>>> the compound PKs we use in some of the Airflow models. We'd > > > > >> have > > > > >>> to > > > > >>>>>> either > > > > >>>>>>>>> write a db migration script on the Airflow side, or add > > support > > > > >>> for > > > > >>>>>>>>> compound keys to FAB (I recently became a maintainer of the > > > > >>>> project, > > > > >>>>>> so I > > > > >>>>>>>>> could help with that) > > > > >>>>>>>>> > > > > >>>>>>>>> The only downside of FAB is that it's not as mature as > > > > >> something > > > > >>>> like > > > > >>>>>>>>> Django, but porting to Django would surely be much more > work. > > > > >>>>>>>>> > > > > >>>>>>>>> Then there's the flask-security suite, but that looks like > a > > > > >> bit > > > > >>>> of a > > > > >>>>>>>>> patchwork to me, I guess we can pick and choose which we > want > > > > >> to > > > > >>>> use. > > > > >>>>>>>>> > > > > >>>>>>>>> Max > > > > >>>>>>>>> > > > > >>>>>>>>> On Mon, Jun 12, 2017 at 12:50 PM, Dan Davydov < > > > > >>>>>>>>> [email protected]> wrote: > > > > >>>>>>>>> > > > > >>>>>>>>>> Looks good to me in general, thanks for putting this > > together! > > > > >>>>>>>>>> > > > > >>>>>>>>>> I think the ability to integrate with external RBAC > systems > > > > >> like > > > > >>>>> LDAP > > > > >>>>>> is > > > > >>>>>>>>>> important (i.e. the Airflow DB should not be decoupled > with > > > > >> the > > > > >>>> RBAC > > > > >>>>>>>>>> database wherever possible). > > > > >>>>>>>>>> > > > > >>>>>>>>>> I wouldn't be too worried about the permissions about > > > > >> refreshing > > > > >>>>>> DAGs, as > > > > >>>>>>>>>> far as I know this functionality is no longer required > with > > > > >> the > > > > >>>> new > > > > >>>>>>>>>> webservers which reload state periodically, and will > > certainly > > > > >>> be > > > > >>>>>> removed > > > > >>>>>>>>>> when we have a better DAG consistency story. > > > > >>>>>>>>>> > > > > >>>>>>>>>> I think it would also be good to think about this > > > > >>>>>> proposal/implementation > > > > >>>>>>>>>> and how it applied in the API-driven world (e.g. when > > > > >> webserver > > > > >>>> hits > > > > >>>>>> APIs > > > > >>>>>>>>>> like /clear on behalf of users instead of running commands > > > > >>> against > > > > >>>>> the > > > > >>>>>>>>>> database directly). > > > > >>>>>>>>>> > > > > >>>>>>>>>> On Mon, Jun 12, 2017 at 11:12 AM, Bolke de Bruin < > > > > >>>> [email protected] > > > > >>>>>> > > > > >>>>>>>>>> wrote: > > > > >>>>>>>>>> > > > > >>>>>>>>>>> Will respond but im traveling at the moment. Give me a > few > > > > >>> days. > > > > >>>>>>>>>>> > > > > >>>>>>>>>>> Sent from my iPhone > > > > >>>>>>>>>>> > > > > >>>>>>>>>>>> On 12 Jun 2017, at 13:39, Chris Riccomini < > > > > >>>> [email protected]> > > > > >>>>>>>>>> wrote: > > > > >>>>>>>>>>>> > > > > >>>>>>>>>>>> Hey all, > > > > >>>>>>>>>>>> > > > > >>>>>>>>>>>> Checking in on this. We spent a good chunk of time > > thinking > > > > >>>> about > > > > >>>>>>>>> this, > > > > >>>>>>>>>>> and > > > > >>>>>>>>>>>> want to move forward with it, but want to make sure > we're > > > > >> all > > > > >>> on > > > > >>>>> the > > > > >>>>>>>>>> same > > > > >>>>>>>>>>>> page. > > > > >>>>>>>>>>>> > > > > >>>>>>>>>>>> Max? Bolke? Dan? Jeremiah? > > > > >>>>>>>>>>>> > > > > >>>>>>>>>>>> Cheers, > > > > >>>>>>>>>>>> Chris > > > > >>>>>>>>>>>> > > > > >>>>>>>>>>>> On Thu, Jun 8, 2017 at 1:49 PM, kalpesh dharwadkar < > > > > >>>>>>>>>>>> [email protected]> wrote: > > > > >>>>>>>>>>>> > > > > >>>>>>>>>>>>> Hello everyone, > > > > >>>>>>>>>>>>> > > > > >>>>>>>>>>>>> As you all know, currently Airflow doesn’t have a > > built-in > > > > >>> Role > > > > >>>>>>>>> Based > > > > >>>>>>>>>>>>> Access Control(RBAC) capability. It does provide very > > > > >>> limited > > > > >>>>>>>>>>>>> authorization capability by providing admin, > > data_profiler, > > > > >>> and > > > > >>>>>> user > > > > >>>>>>>>>>> roles. > > > > >>>>>>>>>>>>> However, associating these roles to authenticated > > > > >> identities > > > > >>> is > > > > >>>>> not > > > > >>>>>>>>> a > > > > >>>>>>>>>>>>> simple effort. > > > > >>>>>>>>>>>>> > > > > >>>>>>>>>>>>> To address this issue, I have created a design proposal > > for > > > > >>>>>> building > > > > >>>>>>>>>>> RBAC > > > > >>>>>>>>>>>>> into Airflow and simplifying user access management via > > the > > > > >>>>> Airflow > > > > >>>>>>>>>> UI. > > > > >>>>>>>>>>>>> > > > > >>>>>>>>>>>>> The design proposal is located at > > > > >> https://cwiki.apache.org/ > > > > >>>>>>>>>>>>> confluence/display/AIRFLOW/Airflow+RBAC+proposal > > > > >>>>>>>>>>>>> > > > > >>>>>>>>>>>>> Any comments/questions/feedback are much appreciated. > > > > >>>>>>>>>>>>> > > > > >>>>>>>>>>>>> Thanks > > > > >>>>>>>>>>>>> Kalpesh > > > > >>>>>>>>>>>>> > > > > >>>>>>>>>>> > > > > >>>>>>>>>> > > > > >>>>>>>>> > > > > >>>>>>> > > > > >>>>>> > > > > >>>>>> > > > > >>>>> > > > > >>>> > > > > >>> > > > > >>> > > > > >>> > > > > >>> -- > > > > >>> > > > > >>> Joy Gao > > > > >>> Software Engineer > > > > >>> 350 Convention Way, Suite 200 > > > > >>> Redwood City, CA 94063 > > > > >>> Mobile: 669-224-9305 > > > > >>> > > > > >>> Payments partner to the platform economy > > > > >>> > > > > >> > > > > > > > > > > > > > >
