Wednesday 8/2 is perfect. Want to do it like 3-5? I booked a room for 12 people (and video conferencing) at WePay in this time slot. Should allow you to head home easily afterwards. :) That work for you guys?
On Mon, Jul 24, 2017 at 11:01 AM, Maxime Beauchemin < [email protected]> wrote: > The week of the 31st sound good. Wednesday? > > About React we may not need a frontend lib like it (or at least not just > yet). We can talk about it at the meeting. > > Max > > On Fri, Jul 21, 2017 at 12:47 AM, Bolke de Bruin <[email protected]> > wrote: > > > We avoid React for the same reasons as the ASF and use Polymer 2 instead. > > Would that work? > > > > Bolke. > > > > > On 20 Jul 2017, at 19:35, Chris Riccomini <[email protected]> > wrote: > > > > > > Hey Max, > > > > > > Want to come down to WePay? We can set up a zoom for those that want to > > > join online, and record it as well to post for the community. > > > > > > Since Joy is just getting started, and it looks like there's going to > be > > a > > > K8s discussion next week, maybe we can shoot for the week after (the > week > > > of the 31st of July)? Care to float a few times that week? > > > > > > Cheers, > > > Chris > > > > > > On Thu, Jul 20, 2017 at 9:31 AM, Maxime Beauchemin < > > > [email protected]> wrote: > > > > > >> Sounds awesome, count me in! > > >> > > >> * check out the prototype in my fork, I went far enough to hit some > > >> hurdles, try different workarounds. I hooked up the Airflow Bootstrap > > >> template too so that we feel at home in this new UI > > >> * using a single `id` field is a requirement for FAB that airflow > > doesn't > > >> respect (composite pks), either we add the feature to support that in > > FAB, > > >> or we align on the Airflow side and modify the models and add a > > migration > > >> script. This upgrade would require downtime and might be annoying to > the > > >> Airflow community, but could help with db performance a bit (smaller > > >> index)... I probably could be convinced either way but I'm leaning on > > >> improving FAB > > >> * I'm a maintainer for FAB so I can help get stuff through there > > >> * React is in limbo at the ASF for licensing reasons, so no React at > > least > > >> for now > > >> * npm/webpack/ES6, javascript only in `.js` files > > >> * I vote for eslint + eslint-config-airbnb as a set of linting rules > > for JS > > >> * Keep out of apache (for now), this new app ships as its own pypi > > package > > >> `airflow-webserver`, have a period of overlap (maintaining 2 web apps) > > >> before ripping out `airflow/www` from the core package > > >> * You need to get in touch with Marty Kausas, an intern at Airbnb > who's > > >> been working on a Flask blueprint for improved, more personalized > views > > on > > >> DAGs that we were planning on merging into the main branch eventually. > > Some > > >> of Marty's idea and code could be merged into this effort. > > >> > > >> These are ideas on how I would proceed personally on this but > definitely > > >> everything here is up for discussion. > > >> > > >> Let's meet physically at either WePay or Airbnb. Folks from the > > community, > > >> let us know on this thread if you want to be part of this effort, > we'll > > be > > >> happy to include you. > > >> > > >> Thanks, > > >> > > >> Max > > >> > > >> On Wed, Jul 19, 2017 at 7:33 PM, Joy Gao <[email protected]> wrote: > > >> > > >>> Hey everyone, > > >>> > > >>> I recently transferred to Data Infra team here at WePay to focus on > > >>> Airflow-related initiatives. > > >>> > > >>> Given the RBAC design is mostly hashed out, I'm happy to get this > > feature > > >>> off the ground for Q3, starting with converting Airflow to Fab, if > > there > > >>> are no objections. > > >>> > > >>> Cheers, > > >>> Joy > > >>> > > >>> On Thu, Jun 29, 2017 at 7:32 AM, Gurer Kiratli < > > >>> [email protected]> wrote: > > >>> > > >>>> Hey all, > > >>>> > > >>>> We talked about this internally. We would like to work on this > feature > > >>> but > > >>>> given the immediate priorities we are not going to be working on it > in > > >>> Q3. > > >>>> Comes end of Q3 we will reevaluate. Likely scenario is we can work > on > > >> it > > >>>> late Q4 or Q12018. > > >>>> > > >>>> Cheers, > > >>>> > > >>>> Gurer > > >>>> > > >>>> On Tue, Jun 27, 2017 at 8:08 AM, Chris Riccomini < > > >> [email protected]> > > >>>> wrote: > > >>>> > > >>>>> I think FAB sounds like the right approach. Waiting to hear back > with > > >>>> notes > > >>>>> on AirBNB H2 discussion to see if they want to take this up. > > >>>>> > > >>>>> @Gurer, any idea when this will happen? > > >>>>> > > >>>>> On Thu, Jun 22, 2017 at 1:00 AM, Bolke de Bruin <[email protected] > > > > >>>> wrote: > > >>>>> > > >>>>>> One downside I see from FAB is that is does not do Business Role > > >>>> mapping > > >>>>>> to FAB role. I would prefer to create groups in IPA/LDAP/AD and > > >> have > > >>>>> those > > >>>>>> map to FAB roles instead of needing to manage that in FAB. > > >>>>>> > > >>>>>> B. > > >>>>>> > > >>>>>>> On 22 Jun 2017, at 09:36, Bolke de Bruin <[email protected]> > > >>> wrote: > > >>>>>>> > > >>>>>>> Hi Guys, > > >>>>>>> > > >>>>>>> Thanks for putting the thinking in! It is about time that we get > > >>> this > > >>>>>> moving. > > >>>>>>> > > >>>>>>> The design looks pretty sound. One can argue about the different > > >>>> roles > > >>>>>> that are required, but that will be situation dependent I guess. > > >>>>>>> > > >>>>>>> Implementation wise I would argue together with Max that FAB is a > > >>>>> better > > >>>>>> or best fit. The ER model that is being described is pretty much a > > >>> copy > > >>>>> of > > >>>>>> a normal security model. So a reimplementation of that is 1) > > >>>> significant > > >>>>>> duplication of effort and 2) bound to have bugs that have been > > >> solved > > >>>> in > > >>>>>> the other framework. Moreover, FAB does have integration out of > the > > >>> box > > >>>>>> with some enterprisey systems like IPA, ActiveDirectory, and LDAP. > > >>>>>>> > > >>>>>>> So while you argue that using FAB would increase the scope of the > > >>>>>> proposal significantly, but I think that is not true. Using FAB > > >> would > > >>>>> allow > > >>>>>> you to focus on what kind of out-of-the-box permission sets and > > >> roles > > >>>> we > > >>>>>> would need and maybe address some issues that FAB lacks (maybe how > > >> to > > >>>>> deal > > >>>>>> with non web access - ie. in DAGs, maybe Kerberos, probably how to > > >>> deal > > >>>>>> with API calls that are not CRUD). Implementation wise it probably > > >>>>>> simplifies what we need to do. Maybe - using Max’s early POC as an > > >>>>> example > > >>>>>> - we can slowly move over? > > >>>>>>> > > >>>>>>> On a side note: Im planning to hire 2-3 ppl to work on Airflow > > >>> coming > > >>>>>> year. Improvement of Security, Enterprise Integration, Revamp UI > > >> are > > >>> on > > >>>>> the > > >>>>>> todo list. However, this is not confirmed yet as business > > >> priorities > > >>>>> might > > >>>>>> change. > > >>>>>>> > > >>>>>>> Bolke. > > >>>>>>> > > >>>>>>> > > >>>>>>>> On 15 Jun 2017, at 21:45, kalpesh dharwadkar < > > >>>>>> [email protected]> wrote: > > >>>>>>>> > > >>>>>>>> @Dan: > > >>>>>>>> > > >>>>>>>> Thanks for your feedback. I will remove the REFRESH_DAG > > >>> permission. > > >>>>>>>> > > >>>>>>>> @Max: > > >>>>>>>> > > >>>>>>>> Thanks for your response. > > >>>>>>>> > > >>>>>>>> The scope of my proposal was just to add RBAC security feature > > >> to > > >>>>>> Airflow > > >>>>>>>> without replacing any existing frameworks. > > >>>>>>>> > > >>>>>>>> I understand that adopting FAB would serve Airflow better moving > > >>>>>> forward, > > >>>>>>>> however porting Airflow to using FAB significantly increases the > > >>>> scope > > >>>>>> of > > >>>>>>>> the proposal and I don't have the time and expertise to carry > > >> out > > >>>> the > > >>>>>> tasks > > >>>>>>>> in the extended scope. > > >>>>>>>> > > >>>>>>>> Hence, I'm curious to know if there's a plan for Airflow to > > >>> migrate > > >>>> to > > >>>>>> FAB > > >>>>>>>> this year? > > >>>>>>>> > > >>>>>>>> - Kalpesh > > >>>>>>>> > > >>>>>>>> On Mon, Jun 12, 2017 at 6:16 PM, Maxime Beauchemin < > > >>>>>>>> [email protected]> wrote: > > >>>>>>>> > > >>>>>>>>> It would be nice to go with a framework for this. I did some > > >>>>>>>>> experimentation using FlaskAppBuilder to go in this direction. > > >> It > > >>>>>> provides > > >>>>>>>>> auth on different authentication backends out of the box > > >> (oauth, > > >>>>>> openid, > > >>>>>>>>> ldap, registration, ...), generates perms for each view that > > >> has > > >>> an > > >>>>>>>>> @has_access decorator, generates at set of perms for each ORM > > >>> model > > >>>>>> (show, > > >>>>>>>>> edit, delete, add, ...) and enforces it in the CRUD views as > > >> well > > >>>> as > > >>>>>> in the > > >>>>>>>>> generated REST api that you get for free as a byprdoduct of > > >>>> deriving > > >>>>>> FAB's > > >>>>>>>>> models (essentially it's SqlAlchemy with a layer on top). > > >>>>>>>>> > > >>>>>>>>> I started a POC on FAB here a while ago: > > >>>>>>>>> https://github.com/mistercrunch/airflow_webserver at the time > > >> my > > >>>>> main > > >>>>>>>>> motivation was the free/instantaneous REST api. > > >>>>>>>>> > > >>>>>>>>> I think FAB is a decent fit as the porting should be fairly > > >>>>>> straightforward > > >>>>>>>>> (moving the flask views over and deprecating Flask-Admin in > > >> favor > > >>>> of > > >>>>>> FAB's > > >>>>>>>>> crud) though there was a few blockers. From memory I think FAB > > >>>> didn't > > >>>>>> like > > >>>>>>>>> the compound PKs we use in some of the Airflow models. We'd > > >> have > > >>> to > > >>>>>> either > > >>>>>>>>> write a db migration script on the Airflow side, or add support > > >>> for > > >>>>>>>>> compound keys to FAB (I recently became a maintainer of the > > >>>> project, > > >>>>>> so I > > >>>>>>>>> could help with that) > > >>>>>>>>> > > >>>>>>>>> The only downside of FAB is that it's not as mature as > > >> something > > >>>> like > > >>>>>>>>> Django, but porting to Django would surely be much more work. > > >>>>>>>>> > > >>>>>>>>> Then there's the flask-security suite, but that looks like a > > >> bit > > >>>> of a > > >>>>>>>>> patchwork to me, I guess we can pick and choose which we want > > >> to > > >>>> use. > > >>>>>>>>> > > >>>>>>>>> Max > > >>>>>>>>> > > >>>>>>>>> On Mon, Jun 12, 2017 at 12:50 PM, Dan Davydov < > > >>>>>>>>> [email protected]> wrote: > > >>>>>>>>> > > >>>>>>>>>> Looks good to me in general, thanks for putting this together! > > >>>>>>>>>> > > >>>>>>>>>> I think the ability to integrate with external RBAC systems > > >> like > > >>>>> LDAP > > >>>>>> is > > >>>>>>>>>> important (i.e. the Airflow DB should not be decoupled with > > >> the > > >>>> RBAC > > >>>>>>>>>> database wherever possible). > > >>>>>>>>>> > > >>>>>>>>>> I wouldn't be too worried about the permissions about > > >> refreshing > > >>>>>> DAGs, as > > >>>>>>>>>> far as I know this functionality is no longer required with > > >> the > > >>>> new > > >>>>>>>>>> webservers which reload state periodically, and will certainly > > >>> be > > >>>>>> removed > > >>>>>>>>>> when we have a better DAG consistency story. > > >>>>>>>>>> > > >>>>>>>>>> I think it would also be good to think about this > > >>>>>> proposal/implementation > > >>>>>>>>>> and how it applied in the API-driven world (e.g. when > > >> webserver > > >>>> hits > > >>>>>> APIs > > >>>>>>>>>> like /clear on behalf of users instead of running commands > > >>> against > > >>>>> the > > >>>>>>>>>> database directly). > > >>>>>>>>>> > > >>>>>>>>>> On Mon, Jun 12, 2017 at 11:12 AM, Bolke de Bruin < > > >>>> [email protected] > > >>>>>> > > >>>>>>>>>> wrote: > > >>>>>>>>>> > > >>>>>>>>>>> Will respond but im traveling at the moment. Give me a few > > >>> days. > > >>>>>>>>>>> > > >>>>>>>>>>> Sent from my iPhone > > >>>>>>>>>>> > > >>>>>>>>>>>> On 12 Jun 2017, at 13:39, Chris Riccomini < > > >>>> [email protected]> > > >>>>>>>>>> wrote: > > >>>>>>>>>>>> > > >>>>>>>>>>>> Hey all, > > >>>>>>>>>>>> > > >>>>>>>>>>>> Checking in on this. We spent a good chunk of time thinking > > >>>> about > > >>>>>>>>> this, > > >>>>>>>>>>> and > > >>>>>>>>>>>> want to move forward with it, but want to make sure we're > > >> all > > >>> on > > >>>>> the > > >>>>>>>>>> same > > >>>>>>>>>>>> page. > > >>>>>>>>>>>> > > >>>>>>>>>>>> Max? Bolke? Dan? Jeremiah? > > >>>>>>>>>>>> > > >>>>>>>>>>>> Cheers, > > >>>>>>>>>>>> Chris > > >>>>>>>>>>>> > > >>>>>>>>>>>> On Thu, Jun 8, 2017 at 1:49 PM, kalpesh dharwadkar < > > >>>>>>>>>>>> [email protected]> wrote: > > >>>>>>>>>>>> > > >>>>>>>>>>>>> Hello everyone, > > >>>>>>>>>>>>> > > >>>>>>>>>>>>> As you all know, currently Airflow doesn’t have a built-in > > >>> Role > > >>>>>>>>> Based > > >>>>>>>>>>>>> Access Control(RBAC) capability. It does provide very > > >>> limited > > >>>>>>>>>>>>> authorization capability by providing admin, data_profiler, > > >>> and > > >>>>>> user > > >>>>>>>>>>> roles. > > >>>>>>>>>>>>> However, associating these roles to authenticated > > >> identities > > >>> is > > >>>>> not > > >>>>>>>>> a > > >>>>>>>>>>>>> simple effort. > > >>>>>>>>>>>>> > > >>>>>>>>>>>>> To address this issue, I have created a design proposal for > > >>>>>> building > > >>>>>>>>>>> RBAC > > >>>>>>>>>>>>> into Airflow and simplifying user access management via the > > >>>>> Airflow > > >>>>>>>>>> UI. > > >>>>>>>>>>>>> > > >>>>>>>>>>>>> The design proposal is located at > > >> https://cwiki.apache.org/ > > >>>>>>>>>>>>> confluence/display/AIRFLOW/Airflow+RBAC+proposal > > >>>>>>>>>>>>> > > >>>>>>>>>>>>> Any comments/questions/feedback are much appreciated. > > >>>>>>>>>>>>> > > >>>>>>>>>>>>> Thanks > > >>>>>>>>>>>>> Kalpesh > > >>>>>>>>>>>>> > > >>>>>>>>>>> > > >>>>>>>>>> > > >>>>>>>>> > > >>>>>>> > > >>>>>> > > >>>>>> > > >>>>> > > >>>> > > >>> > > >>> > > >>> > > >>> -- > > >>> > > >>> Joy Gao > > >>> Software Engineer > > >>> 350 Convention Way, Suite 200 > > >>> Redwood City, CA 94063 > > >>> Mobile: 669-224-9305 > > >>> > > >>> Payments partner to the platform economy > > >>> > > >> > > > > >
