Re: Airflow 2.0

Could we make a push to close some of the open PRs and get them into a 1.10 release before focusing on 2.0? I can make some time to review, if there are other folks that can follow up and merge the PRs. --George On Tue, Dec 19, 2017 at 2:42 PM Chris Riccomini wrote: > > It does make sense to k

Re: Airflow 2.0

> It does make sense to keep SQLite on the other hand a Docker image with all the components might just be as convenient? I formed an actual opinion. :) IMO, SQLite is very convenient and should be left in. > Dropping stuff would lessen the burden of maintenance. There is a lot of cruft that is n

Re: Airflow 2.0

Hi George, It does make sense to keep SQLite on the other hand a Docker image with all the components might just be as convenient? Dropping stuff would lessen the burden of maintenance. There is a lot of cruft that is not used. This reduces the surface for bugs and makes it easier to do the op

Re: Airflow 2.0

I really do not think we should drop sqlite support. We use sqlite for local testing/development; I used it when doing my initial evaluation of Airflow; and there are regular comments in this group about people using sqlite locally for their workflows. It feels like a critical feature for a lot of

Re: Airflow 2.0

We use sqlite for developing Airflow DAGs locally, and doing basic checks for syntax/import errors in our CI/CD pipeline. Alek On Thu, Dec 14, 2017 at 4:19 PM, Chris Riccomini wrote: > Agree with Bolke that it's a good idea, but major work. I will push back on > this for 2.0 mostly due to time

Re: Airflow 2.0

Agree with Bolke that it's a good idea, but major work. I will push back on this for 2.0 mostly due to time concerns. I don't want 2.0 to take months and months to get out. > Btw: what about dropping sqlite support? I'm fine with this, but isn't this really useful for demos? Seems to me that the

Re: Airflow 2.0

That will take quite some work. It is a a good idea but also a major change. Not sure if we should target that. Btw: what about dropping sqlite support? Verstuurd vanaf mijn iPad > Op 14 dec. 2017 om 21:19 heeft Gael Magnan het > volgende geschreven: > > Hi, > > haven't been following much

Re: Airflow 2.0

That will take quite some work. It is a a good idea but also a major change. Not sure if we should target that. Btw: what about dropping sqlite support? Verstuurd vanaf mijn iPad > Op 14 dec. 2017 om 21:19 heeft Gael Magnan het > volgende geschreven: > > Hi, > > haven't been following much

Re: Airflow 2.0

Hi, haven't been following much lately but on the import side of things, isn't Airflow 2 the best moment to change to a pip plugin system for imports of third party stuff? I.E being able to add a new type of credentials, operator etc.. without touching to the airflow code itself or having them in

Re: Airflow 2.0

@Bolke, > Should we, before 2.0, start the graduation from the incubator? No, I'd rather keep them separate. We can certainly start graduation, but I don't want to block 2.0. Can pursue them in parallel. On Thu, Dec 14, 2017 at 11:15 AM, Andy Hadjigeorgiou wrote: > Does it make sense to includ

Re: Airflow 2.0

I'm fine with sensor refactor. Added to Wiki. On Thu, Dec 14, 2017 at 11:16 AM, Chris Riccomini wrote: > @Bolke, > > > Should we, before 2.0, start the graduation from the incubator? > > No, I'd rather keep them separate. We can certainly start graduation, but > I don't want to block 2.0. Can pu

Re: Airflow 2.0

Does it make sense to include sensors.py refactor in 2.0, so we can retire the old import structure easily and support the new sensors package import structure? - Andy On Thu, Dec 14, 2017 at 2:12 PM, Driesprong, Fokko wrote: > Hi all, > > Good initiative. I would be happy to refactor the senso

Re: Airflow 2.0

Hi all, Good initiative. I would be happy to refactor the sensors package. I started on it but it changes a lot, all the imports will break. https://github.com/apache/incubator-airflow/pull/2875 What do you guys think? Cheers, Fokko 2017-12-14 20:09 GMT+01:00 Chris Riccomini : > I have create

Re: Airflow 2.0

Should we, before 2.0, start the graduation from the incubator? B. Sent from my iPhone > On 14 Dec 2017, at 20:09, Chris Riccomini wrote: > > I have created a wiki here: > > https://cwiki.apache.org/confluence/display/AIRFLOW/Airflow+2.0 > > To track features and progress. > > On Thu, Dec

Re: Airflow 2.0

I have created a wiki here: https://cwiki.apache.org/confluence/display/AIRFLOW/Airflow+2.0 To track features and progress. On Thu, Dec 14, 2017 at 11:08 AM, Chris Riccomini wrote: > > Re: #2: Is there a current ticket out for removing the legacy import > style? > > No, I don't think so, but y

Re: Airflow 2.0

> Re: #2: Is there a current ticket out for removing the legacy import style? No, I don't think so, but you can create one! :) On Thu, Dec 14, 2017 at 11:06 AM, Andy Hadjigeorgiou wrote: > This sounds great, something I'd like to see updated for 2.0 release (or > before) is the Airflow document

Re: Airflow 2.0

This sounds great, something I'd like to see updated for 2.0 release (or before) is the Airflow documentation ( http://airflow.readthedocs.io/en/latest/installation.html). It seems that updating the repo does not update this site - and giv

Re: Airflow 2.0

> Kick out pickle? Yea, totally. > Remove all their deprecated features? I'm all for removing deprecated features. Need a full explicit list, though. :) On Thu, Dec 14, 2017 at 11:01 AM, Bolke de Bruin wrote: > Kick out pickle? > Remove all their deprecated features? > > Verstuurd vanaf mijn

Re: Airflow 2.0

Kick out pickle? Remove all their deprecated features? Verstuurd vanaf mijn iPad > Op 14 dec. 2017 om 19:55 heeft Chris Riccomini het > volgende geschreven: > > Hey all, > > With 1.9.0 wrapping up soon (hopefully), there's been some discussion on > the having the next release be Airflow 2.0 (

Re: Airflow 2.0

nces can be uniquely attributed to dag runs. > > >>>> > > >>>> > > >>>> + 1 > > >>>> > > >>>> > > >>>> Also, I believe xcoms would need to be addressed in the same way at > > the > &

Re: Airflow 2.0

ssues.apache.org/jira/browse/AIRFLOW-642 > >>>> > >>>> > >>>> I believe this would be implemented for all supported back-ends, not > >> just > >>>> PostgreSQL. > >>>> > >>>> > >>>>

Re: Airflow 2.0

org/jira/browse/AIRFLOW-642 >>>> >>>> >>>> I believe this would be implemented for all supported back-ends, not >> just >>>> PostgreSQL. >>>> >>>> >>>> Cheers, >>>> Luke Maycock >>>> O

Re: Airflow 2.0

; > > Cheers, > > > Luke Maycock > > > OLIVER WYMAN > > > luke.mayc...@affiliate.oliverwyman.com<mailto:luke. > > > mayc...@affiliate.oliverwyman.com><mailto:luke.maycock@ > > > affiliate.oliverwyman.com> > > > www.oliverwyman.

Re: Airflow 2.0

iverwyman.com<http://www.oliverwyman.com> > www.oliverwyman.com/> > > > > > > > > > > From: Arunprasad Venkatraman mailto:arp...@uber.com>> > > Sent: 21 November 2016 18:16 > > To: dev@airflow.incubator.apache.org<ma

Re: Airflow 2.0

/> > > > > > From: Arunprasad Venkatraman mailto:arp...@uber.com>> > Sent: 21 November 2016 18:16 > To: dev@airflow.incubator.apache.org<mailto:dev@airflow. > incubator.apache.org> > Subject: Re: Airflow 2.0 > > Add FK to dag_run to the task_in

Re: Airflow 2.0

n mailto:arp...@uber.com>> Sent: 21 November 2016 18:16 To: dev@airflow.incubator.apache.org<mailto:dev@airflow.incubator.apache.org> Subject: Re: Airflow 2.0 Add FK to dag_run to the task_instance table on Postgres so that task_instances can be uniquely attributed to dag runs. Ensure sc

Re: Airflow 2.0

verwyman.com/> From: Arunprasad Venkatraman Sent: 21 November 2016 18:16 To: dev@airflow.incubator.apache.org Subject: Re: Airflow 2.0 > Add FK to dag_run to the task_instance table on Postgres so that task_instances can be uniquely attributed to dag runs. > Ensure scheduler can be run

Re: Airflow 2.0

elery's > > > >> concurrency parameter at the airflow server level is too > > coarse-grained > > > as > > > >> it forces all workers to be the same, and does not allow proper > > resource > > > >> management when different workflow tasks

Re: Airflow 2.0

workflow tasks have different resource > > >> requirements thus hurting utilization (a worker could run 8 parallel > > tasks > > >> with small memory footprint, but only 1 task with large memory > footprint > > >> for instance). > > >> > > >&g

Re: Airflow 2.0

t;> With best regards, > >> > >> Sergei. > >> > >> > >> On Mon, Nov 21, 2016 at 2:00 PM Ryabchuk, Pavlo < > >> ext-pavlo.ryabc...@here.com> > >> wrote: > >> > >> > -1. We extremely rely on data profiling, as a pipeline health

Re: Airflow 2.0

pavlo.ryabc...@here.com> >> wrote: >> >> > -1. We extremely rely on data profiling, as a pipeline health monitoring >> > tool >> > >> > -Original Message- >> > From: Chris Riccomini [mailto:criccom...@apache.org] >> > Sent: Saturday, November 19

Re: Airflow 2.0

ealth monitoring > > tool > > > > -Original Message- > > From: Chris Riccomini [mailto:criccom...@apache.org] > > Sent: Saturday, November 19, 2016 1:57 AM > > To: dev@airflow.incubator.apache.org > > Subject: Re: Airflow 2.0 > > > > >

Re: Airflow 2.0

t; >> for instance). > >> > >> With best regards, > >> > >> Sergei. > >> > >> > >> On Mon, Nov 21, 2016 at 2:00 PM Ryabchuk, Pavlo < > >> ext-pavlo.ryabc...@here.com> > >> wrote: > >> > >

Re: Airflow 2.0

gt; >> > >> Sergei. > >> > >> > >> On Mon, Nov 21, 2016 at 2:00 PM Ryabchuk, Pavlo < > >> ext-pavlo.ryabc...@here.com> > >> wrote: > >> > >> > -1. We extremely rely on data profiling, as a pipeline health > monitori

Re: Airflow 2.0

>> With best regards, >> >> Sergei. >> >> >> On Mon, Nov 21, 2016 at 2:00 PM Ryabchuk, Pavlo < >> ext-pavlo.ryabc...@here.com> >> wrote: >> >> > -1. We extremely rely on data profiling, as a pipeline health monitoring >> > tool

Re: Airflow 2.0

ssage- > > From: Chris Riccomini [mailto:criccom...@apache.org] > > Sent: Saturday, November 19, 2016 1:57 AM > > To: dev@airflow.incubator.apache.org > > Subject: Re: Airflow 2.0 > > > > > RIP out the charting application and the data profiler >

Re: Airflow 2.0

th monitoring > tool > > -Original Message- > From: Chris Riccomini [mailto:criccom...@apache.org] > Sent: Saturday, November 19, 2016 1:57 AM > To: dev@airflow.incubator.apache.org > Subject: Re: Airflow 2.0 > > > RIP out the charting application and the data

RE: Airflow 2.0

-1. We extremely rely on data profiling, as a pipeline health monitoring tool -Original Message- From: Chris Riccomini [mailto:criccom...@apache.org] Sent: Saturday, November 19, 2016 1:57 AM To: dev@airflow.incubator.apache.org Subject: Re: Airflow 2.0 > RIP out the chart

Re: Airflow 2.0

Hi, Like we have an admin panel,where we can configure the database connections and query them . Similarly based on the executor backend chosen, some information should be provided. Like for Airflow + rabbit Mq + Celery backend, if rabbit mq goes down, it keeps on showing the message that task ha

Re: Airflow 2.0

I feel a lot of changes happen to areas of the code shared by both scheduler and webserver, such as models. Any time we have changes to these shared areas, we will need to release the scheduler as well. Also, it's not clear to me (out of ignorance perhaps) how the above would speed up releasing.

Re: Airflow 2.0

Totally agree on all your points Sid. My feeling is that at the moment the most critical thing for the project is to get a release out and get to a steady pace of high quality releases. Somehow breaking down the package seem to me like it would really help with the release process. Maybe an idea

Re: Airflow 2.0

David https://issues.apache.org/jira/browse/AIRFLOW-558 (i.e. http s://github.com/apache/incubator-airflow/pull/1830 ) Is on my plate.. have already gone through many rounds of reviews, testing, and fixes with the submitter and does not need to wait till 2.0. We should be able to merge it soon. BTW

Re: Airflow 2.0

> RIP out the charting application and the data profiler Yes please! +1 On Fri, Nov 18, 2016 at 2:41 PM, Maxime Beauchemin wrote: > Another point that may be controversial for Airflow 2.0: RIP out the > charting application and the data profiler. Even though it's nice to have > it there, it's ju

Re: Airflow 2.0

Another point that may be controversial for Airflow 2.0: RIP out the charting application and the data profiler. Even though it's nice to have it there, it's just out of scope and has major security issues/implications. I'm not sure how popular it actually is. We may need to run a survey at some p

Re: Airflow 2.0

Using FAB's Model, we get pretty much all of that (REST API, auth/perms, CRUD) for free: http://flask-appbuilder.readthedocs.io/en/latest/quickhowto.html?highlight=rest#exposed-methods I'm pretty intimate with FAB since I use it (and contributed to it) for Superset/Caravel. All that's needed is t

Re: Airflow 2.0

> It may be doable to run this as a different package `airflow-webserver`, an > alternate UI at first, and to eventually rip out the old UI off of the main > package. This is the same strategy that I was thinking of for AIRFLOW-85. You can build the new UI in parallel, and then delete the old one

Re: Airflow 2.0

A multi-tenant UI with composable roles on top of granular permissions. Migrating from Flask-Admin to Flask App Builder would be an easy-ish win (since they're both Flask). FAB Provides a good authentication and permission model that ships out-of-the-box with a REST api. Suffice to define FAB mode

Re: Airflow 2.0

(+1 on AIRFLOW-85.. but that's my ticket from a while ago :) Also, killing off the deprecated __init__.py magic in hooks/operators, and going with standard imports. On Fri, Nov 18, 2016 at 1:32 PM, Chris Riccomini wrote: > Full-fledged REST API (that the UI also uses) would be great in 2.0. > >

Re: Airflow 2.0

Full-fledged REST API (that the UI also uses) would be great in 2.0. On Fri, Nov 18, 2016 at 6:26 AM, David Kegley wrote: > Hi All, > > We have been using Airflow heavily for the last couple months and it’s been > great so far. Here are a few things we’d like to see prioritized in 2.0. > > 1) Ro

Re: Airflow 2.0

Hi All, We have been using Airflow heavily for the last couple months and it’s been great so far. Here are a few things we’d like to see prioritized in 2.0. 1) Role based access to DAGs: We would like to see better role based access through the UI. There’s a related ticket out there but it hasn