Re: Graduation resolution passed - Airflow is a TLP

Amazing! Great work all! On Thu, Dec 20, 2018, 13:13 Jakob Homan Hey all- >The Board minutes haven't been published yet (probably due to > Holiday-related slowness), but I can see through the admin tool that > our Graduation resolution was approved yesterday at the meeting. > Airflow is the

Re: [VOTE] Graduate the Apache Airflow as a TLP

cope of > responsibility of the Apache Airflow Project; and be it further > > RESOLVED, that the persons listed immediately below be and > hereby are appointed to serve as the initial members of the > Apache Airflow Project: > > * Alex Guziel > * Alex Van Boxel > * Arthu

Re: Will Airflow 2.0.0 support Python 2.7?

For what it's worth I do not think you are wrong. Given that we are definitely depending on a large ecosystem of packages, some of which might drop support next year, it might make sense to at least announce deprecation in 2.0 and deprecate in 3.0. 3.0 could happen early 2020 though. Best,

Re: Pinning dependencies for Apache Airflow

t; its own. > > > > > > On Thu, Oct 4, 2018 at 9:20 AM Jarek Potiuk > > > wrote: > > > > > > > Great that this discussion already happened :). Lots of useful things > > in > > > > it. And yes - it means pinning in requirement.

Re: Pinning dependencies for Apache Airflow

Hi Jarek, I will +1 the discussion Dan is referring to and George's advice. I just want to double check we are talking about pinning in requirements.txt only. This offers the ability to pip install -r requirements.txt pip install --no-deps airflow For a guaranteed install which works. Several

Re: [VOTE][RESULT] Replace with Gitter with Slack?

gt;> Op zo 9 sep. 2018 03:47 schreef Sid Anand : > > >> > > >>> +1: 12 votes > > >>> > > >>> - Sid Anand > > >>> - Steve Carpenter > > >>> - Beau Barker > > >>> - Marc Bollinger > > >>

Re: Apache Spark Interfering with Airflow Jira/PRs ??

I am guessing Holden is testing the JIRA bot/integration, since some of the JIRA's are being assigned to her magical unicorn 濾. Holden? Best, Arthur On Sun, Sep 2, 2018, 12:48 Kaxil Naik wrote: > I am getting that too. Tons of emails as well about the same. Not sure the > reason. > > On Sun,

Re: Python 3.6 Support for Airflow 1.10.0

Given that Python 2.7 EOL is slated for January 1st 2020, we should probably ensure that the early releases of 2019 are still 2.7 compatible. Beyond this, I think we can also be responsible security wise and help nudge people towards 3. Best, Arthur On Tue, Aug 28, 2018 at 10:28 AM Bolke de

Re: [RESULT][VOTE] Release Airflow 1.10.0

Done for Bolke, Fokko and kaxil. Best, Arthur On Sun, Aug 26, 2018 at 3:08 AM Driesprong, Fokko wrote: > Gentle ping! Would be awesome to get 1.10 on Pypi :-) > > Op wo 22 aug. 2018 om 23:43 schreef Naik Kaxil : > > > Mine is "kaxil" > > > > On 22/08/2018, 16:18, "Bolke de Bruin" wrote: > >

Re: [DISCUSS] AIP - Time for Airflow Improvement Proposals?

+1 On Sun, Jul 15, 2018, 20:12 Maxime Beauchemin wrote: > +1 > > On Tue, Jul 10, 2018 at 1:09 PM Sid Anand wrote: > > > +1 > > > > On Tue, Jul 10, 2018 at 1:02 PM George Leslie-Waksman > > wrote: > > > > > +1 > > > > > > On Tue, Jul 10, 2018 at 11:50 AM Jakob Homan > wrote: > > > > > > >

Re: Adding moderators to the email lists for an incubating project

Sid, Count me in. My apache user is 'arthur' Best, Arthur On Fri, Jun 1, 2018 at 4:28 PM Sid Anand wrote: > Which committers would like to be added as dev and commits list moderators > for Airflow? If interested, let me know on this thread and I will open an > infra ticket. I get between 5-10

Re: Interesting things about how to know it's a DAG file

s themselves? > > Alternatively, there could be a single entry point (ie, index.py) from > which all the DAGs are instantiated. There's probably some complexity in > making that work with the multi-process scheduler model, but doesn't seem > insurmountable. > > On Thu, May 10, 2018 at 10:3

Re: Interesting things about how to know it's a DAG file

Hi Song, I agree that this is not ideal, but it is difficult to do otherwise without parsing/executing the Python code. Note that an import from airflow should be enough, or DAG in a comment. I think we are open to other solutions, if anyone on the list has better ideas. Best, Arthur On

Re: Subdag tasks unable to start

Glad it worked out! Best, Arthur On Wed, May 9, 2018 at 3:26 AM Victor Noagbodji < vnoagbo...@amplify-nation.com> wrote: > Thanks Arthur. > > I went with having two DAGs as you suggested. For anyone reading, that's > what worked for me. > > > On May 8, 2018,

Re: Subdag tasks unable to start

Hi Victor, Usually, we expect the Schedules to match with Subdags, I am not sure Airflow supports heterogeneous schedules like this. What you could have is one Triggering DAG, and a second DAG that it triggers at the right interval. Would you be able to explain further what you mean by simulate

Re: Problem with SparkSubmit

No excuse necessary, we are glad you found the issue! Best, Arthur On Tue, May 8, 2018 at 8:06 AM Anton Mushin wrote: > I found mistake in my connection configuration. > The problem is not relevant. Excuse me for troubling. > > Best Regards, > Anton > > -Original

Re: Airflow tree view display issue

Hi Larry, Unfortunately, embedded images do not make through the mailing list. Would you be able to send a link to your picture, via the hosting service of your choice so we can help diagnose? Best, Arthur On Tue, May 8, 2018 at 8:06 AM Wang, Larry wrote: > Hi Airflow

Re: Airflow 1.9.1 expected release?

Hi, We are currently looking to release 1.10 soon-ish. I do not think we plan on releasing 1.9.1 unless there is a vulnerability or critical bug. Best, Arthur On Wed, Apr 18, 2018 at 9:13 AM, Zsolt Tóth wrote: > Hey all, > > is there a planned (approximate) release

Re: Heads up: OMG PR with massive changes

I will take a look this afternoon. On Thu, Apr 12, 2018, 12:00 Bolke de Bruin wrote: > Hi All, > > To fix license headers I have created a PR. Spoiler: it is massive. And, > worse, I want to push it pretty quickly. > > It is incubating at: > >

Re: Readthedocs

Done, Sid! On Tue, Apr 10, 2018 at 3:34 PM, Sid Anand wrote: > Kaxil, > The same request for you. My user id is r39132. > > -s > > On Tue, Apr 10, 2018 at 5:21 AM, Driesprong, Fokko > wrote: > > > Thanks Kaxil! > > > > Cheers, Fokko > > > > 2018-04-10

Re: Podling Report Reminder - April 2018

Thanks David! I will update our part of the report. Best, Arthur On Thu, Apr 5, 2018 at 9:57 AM, David Fisher wrote: > Hi - > > Answering myself. I see that the licensing issue is resolved and you are > waiting for a release. A clearer statement might be: > > "Once we make a

Re: Podling Report Reminder - April 2018

Hi David, One of the libraries we were using pulled in a dependency which itself depended on code licensed under the GPL. The issue is discussed here: https://issues.apache.org/jira/browse/LEGAL-362 And tracked in the Apache Airflow (incubating) project here:

DAG Level permissions (was Re: RBAC Update)

(Creating a new thread) Hi Max, I was just wondering about this. There are definite use cases for people having only view access to some DAGs, mostly for monitoring. I want to know what the upstream DAGs are doing, but maybe I don't need clear/run access. I feel like the granular operation

Re: RBAC Update

Thanks a lot for all the effort Joy! Great job! Best, Arthur On Fri, Mar 23, 2018 at 6:45 PM, Joy Gao wrote: > Hey guys! > > The RBAC UI has > been merged to master. I'm looking forward to early adopters' feedback and >

Re: Airflow Documentation - Readthedocs

Done. Thanks for the help Andy! On Fri, Feb 9, 2018 at 6:56 AM, Andy Hadjigeorgiou <andyxha...@gmail.com> wrote: > My username is andyxhadji, email is andyxha...@gmail.com. > > Thanks for the help Arthur! > > - Andy > > On Fri, Feb 9, 2018 at 9:46 AM, Arthur Wiedme

Re: Airflow Documentation - Readthedocs

Someone needs to create an account on the site, I can then add them as a maintainer for the project. Best, Arthur On Sun, Feb 4, 2018 at 8:14 AM, Bolke de Bruin wrote: > Perfectly fine to me. It would be highly appreciated if this could be > picked up so it can be made opart

Re: PR Review request for url_prefix

n. I think this is the right approach. > > As the Rest API is connected to the webserver at the moment,I do think it > is also correct to move the API endpoints indeed. If we think this should > be managed independently it should be mounted separately. > > Cheers > Bolke &

Re: PR Review request for url_prefix

Bolke, Wouldn't application_root cause an issue with cookies ( https://stackoverflow.com/questions/18967441/add-a-prefix-to-all-flask-routes) since Gaetan did not want to move the API endpoint, or do you recommend the API endpoint to be moved as well in this scenario? Best, Arthur On Sat, Jan

Re: PR Review request for url_prefix

Gaetan, I did a quick review. Overall the changes look good. A couple of things to check. Best, Arthur On Sat, Jan 6, 2018 at 1:46 AM, Gaetan Semet wrote: > Hello > > Can a core developer look at my pull request ? > https://github.com/apache/incubator-airflow/pull/2723 > >

Re: Airflow 1.9.0 is released

Woohoo!  Thanks Chris! On Tue, Jan 2, 2018 at 2:40 PM, Chris Riccomini wrote: > Dear Airflow community, > > Airflow 1.9.0 was just released. > > The source release as well as the binary "sdist" release are available > here: > >

Re: Switching minimum support version from python 3.4 -> 3.5

+1 as well. On Tue, Dec 19, 2017 at 10:19 AM, Dan Davydov < dan.davy...@airbnb.com.invalid> wrote: > Sounds good to me > > On Tue, Dec 19, 2017 at 10:18 AM Chris Riccomini > wrote: > > > :thumbsup: > > > > On Tue, Dec 19, 2017 at 5:19 AM, Bolke de Bruin

Re: Data lineage and data portal

Likewise! Best, Arthur On Mon, Nov 27, 2017 at 11:57 AM, Alison Stanton < astan...@bankofknowledge.net> wrote: > I'd like to be kept informed. > > Alison Stanton > > On Mon, Nov 27, 2017 at 1:53 PM, Laura Lorenz > wrote: > > > We're definitely looking for something

Re: [VOTE] Airflow 1.9.0rc2

The current version of Airflow we have running at Netflix (my new employer) seems to be doing OK, and it's running off of master. +1 binding On Nov 16, 2017 09:04, "Chris Riccomini" wrote: > Hey all, > > I'm going to let this vote go bit longer. > > Dan, Arthur, anyone

Re: [VOTE] Airflow 1.9.0rc1

res,celery,rabbitmq]=={{version}}' > > > > $ pip install 'git+git://github.com/apache/ > incubator-airflow@{{version}}#egg=apache-airflow[postgres,celery,rabbitmq] > ' > > > > Alek > > ​ > > > > On Thu, Nov 9, 2017 at 3:53 PM, Arthur Wiedmer <arthur.wi

Re: [VOTE] Airflow 1.9.0rc1

ussion. PyPi is not an official > > apache channel, so in theory we could put anything on PyPI. I also think > > (didn’t confirm) pip doesn’t upgrade to RC/beta etc. > > > > Any thoughts? > > > > Bolke. > > > > > On 9 Nov 2017, at 15:53, Arthur W

Re: [VOTE] Airflow 1.9.0rc1

Hi Alek, Technically, we cannot release a distribution on PyPI until we have voted on a release. And here usually a release artifact. It is a little convoluted in the case of Python, but we are getting the hang of it. That said, installing from a git reference is a possibility too if you want

Re: Multiple similar DAGs

Hi Michael, You could use a simple loop Have a single DAG python script which parses a dictionary with the different patterns and parameters. #imports ... dag_info = { 'dag_1': {'schedule_interval': value1}, 'dag_2': {'schedule_interval': value2},

Re: Moving on to Lyft

Congratulations, Max! On Thu, Oct 26, 2017 at 8:25 AM, Chris Riccomini wrote: > Congrats, Max! > > On Thu, Oct 26, 2017 at 7:32 AM, Andy Hadjigeorgiou > wrote: > > > Congrats! Best of luck in your new role! > > > > On Thu, Oct 26, 2017 at 10:28

Re: 1.9.0 test branch has been cut

Are you signed in? I have the same issue if I forget to sign in at the top right. Best, Arthur On Wed, Sep 13, 2017 at 12:59 PM, Ash Berlin-Taylor < ash_airflowl...@firemirror.com> wrote: > Unless I'm missing something I don't think I have permission - at least > I'm not seeing any Edit links

Re: Completed tasks not being marked as completed

I have noticed the issue more prominently with Chrome after v54, I was wondering if this was related at all. Somehow, airflow seems kinder on Firefox. Does anyone else have a similar experience? Best, Arthur On Fri, Aug 4, 2017 at 12:02 PM, Marc Weil wrote: > This is on

Re: Resolving the GPL Licensing issues with Unidecode

f this,I don't see any issues for Till to upgrade to > 1.8+, unless they prepare own Airflow release which includes all the > deps to distribute it across own company network. > > -- > ,,,^..^,,, > > On Fri, Aug 4, 2017 at 3:23 AM, Arthur Wiedmer <arthur.wied...@gmail.com

Re: Resolving the GPL Licensing issues with Unidecode

I would qualify that as a blocker. Best, Arthur On Aug 3, 2017 16:36, "Maxime Beauchemin" wrote: > Hey, how does this affect the current release(s) taking place? > > Max > > On Thu, Aug 3, 2017 at 8:57 AM, Bolke de Bruin wrote: > > > Oh that is a

Re: Deploy procedure for new/modify dags

Hi Germain, As long as the structure of the DAG is not changed (tasks are the same and the dependency graph does not change), there should be no need to restart anything. The scheduler only needs the structure of the DAG to send the right message to celery. Essentially the message tells the

Re: R Support

Hi, Just to share a little bit about our experience. The R parallelism packages have a tendency to use as many cores as they see available by default if you do not set the parallelism explicitly. We have seen R happily fork and create 64 processes on some machines because the AWS box was

Re: Documentation Versioning

The documentation is static files hosted on an apache website. The docs on readthedocs are versioned though : http://airflow.readthedocs.io/en/latest/ Best, Arthur On Thu, May 18, 2017 at 11:37 AM, Bolke de Bruin wrote: > Completely agree, but we need some help here I

Re: Airflow Testing Library

Hi, I would love to see if we can contribute some of the work we have done internally at Airbnb to support some testing of DAGs. We have a long ways to go though :) Best, Arthur On Tue, May 9, 2017 at 12:34 PM, Sam Elamin wrote: > Thanks Gerard and Laura, I have

Re: dag file processing times

What if the DAG actually depends on configuration that only exists in a database and is retrieved by the Python code generating the DAG? Just asking because we have this case in production here. It is slowly changing, so still fits within the Airflow framework, but you cannot just watch a file...

Re: PTAL: Airflow 2017 April Podling Report

I added the following to the report, as it seemed like it was a required question : How would you assess the podling's maturity?Please feel free to add your own commentary. [ ] Initial setup [ ] Working towards first release [ ] Community building [X] Nearing graduation [ ] Other:The Airflow

Re: [RESULT][VOTE]Release Airflow 1.8.0 based on Airflow 1.8.0rc5

re of regenerating the docs + pypi upload, just > > let > > > me > > > >> know when > > > >> > > > >> Max > > > >> > > > >> On Fri, Mar 17, 2017 at 5:20 PM, Dan Davydov < > dan.davy...@airbnb.c

Re: [VOTE] Release Airflow 1.8.0 based on Airflow 1.8.0rc5

+1 (binding) Thanks again for steering us through Bolke. Best, Arthur On Sun, Mar 12, 2017 at 9:59 PM, Bolke de Bruin wrote: > Dear All, > > Finally, I have been able to make the FIFTH RELEASE CANDIDATE of Airflow > 1.8.0 available at: https://dist.apache.org/repos/ >

Re: [VOTE] Release Airflow 1.8.0 based on Airflow 1.8.0rc4

+1 (binding) Thanks again for all the work! Best, Arthur On Fri, Feb 17, 2017 at 4:46 PM, Jeremiah Lowin wrote: > +1 (binding) many thanks for all your work on this Bolke! > > On Fri, Feb 17, 2017 at 7:10 PM Jayesh Senjaliya > wrote: > > +1 (

Re: Flow-based Airflow?

>From our own data warehouse, there are definitely cases where knowing that the data is there is not enough. While I agree that ideally the dependency in data should be explicit, the current dependency engine allows you to compress some of the data dependencies by using the task dependencies.

Re: Experiences with 1.8.0

Maybe we can start with " .with_hint(TI, 'USE INDEX (PRIMARY)', dialect_name='mysql')" and see if other databases exhibit the same query plan issue ? Best, Arthur On Mon, Jan 23, 2017 at 3:27 PM, Chris Riccomini wrote: > With this patch: > > $ git diff > diff --git

Re: Experiences with 1.8.0

Chris, Just double checking, you mean more than 15 seconds not 15 minutes, right? Best, Arthur On Mon, Jan 23, 2017 at 12:27 PM, Chris Riccomini wrote: > Hey all, > > I've upgraded on production. Things seem to be working so far (only been an > hour), but I am seeing

Re: Last minute open meetup speaking slot

you want to fill in, that would be great. Regards, --George On Tue, Jan 10, 2017 at 4:56 PM Arthur Wiedmer <arthur.wied...@gmail.com> wrote: > George, > > How long are the time slots? > > I might be able to put something together about some of the frameworks we > have been usi

Re: Last minute open meetup speaking slot

George, How long are the time slots? I might be able to put something together about some of the frameworks we have been using at Airbnb on top of Airflow. Best, Arthur On Tue, Jan 10, 2017 at 4:36 PM, George Leslie-Waksman < geo...@cloverhealth.com.invalid> wrote: > One of the speakers for

Re: [VOTE]: Using fractional seconds

Hi all, I was the main proponent of option 2, mostly because I could not see a specific situation where sub second precision was needed for this. However, I feel that we have heard from the community that there are use cases out there. I agree with Bolke's analysis of the increased operational

Re: Cloud Provider grouping into Plugins

Sounds reasonable to me. +1 We have been meaning to refactor the AWS operators anyway, we could take advantage of this to reorganize the repo a little bit. That makes me think that we have not yet decided of a flow to enable work towards say 2.0 vs 1.8. We might want to start thinking about

Re: +1 on PRs!

Another way to do this would be to use the reactions on Github. Use the  to upvote an issue or PR you care about. And reserve the +1s for Committer Code review. On top of that, it makes it easier to tally the votes. Best, Arthur On Tue, Oct 11, 2016 at 2:31 PM, siddharth anand

Re: Call to Committers : PR Clean up Duty : ETA Nov 15

The deadline is November 15th, not October 15th, right? I am finishing a big unrelated project here by the end of the week and will take some time for Airflow afterwards. Just so Sid is reassured ;p Best, Arthur On Sat, Oct 8, 2016 at 9:59 AM, siddharth anand wrote: >

Re: Name the DAG and the file for DAG ?

Hi Jason, Naming things with dots is notoriously tricky in Python, and filenames with dots often lead to import errors. Because of the way dags and subdags are handled internally, '.'s (dots) are not supported in DAG names. May I recommend the use of underscores? Best, Arthur On Sun, Sep 4,

Re: airflow API using java code

Hi, I am unsure what you mean: - Do you mean that the code you have to execute is written in Java? (If so you could either use a BashOperator or write a custom operator deriving it) - Do you want to call Airflow from Java? If so I am not sure what your use case is here, but Runtime.exec sounds

Re: Splitting models.py

dependent on Airbnb) before breaking too much stuff might not be a bad idea. Best, Arthur On Thu, Jun 2, 2016 at 9:50 AM, Arthur Wiedmer <art...@apache.org> wrote: > I'd love to do it. Actually this + refactoring the core.py tests would be > amazing. > > But the amount of havok to fi

Re: Splitting models.py

I'd love to do it. Actually this + refactoring the core.py tests would be amazing. But the amount of havok to fix stuff afterwards, including temporary compatibility adjustments would require maybe a temporary lock of quiet time on the models. It is hard to catch all of the added changes in the

Re: Airflow installation issue

Actually, I am pretty sure that we do not support Python 2.6 (I am almost positive that some operators use features like OrderedDict to capture schemas, which was only introduced in 2.7). I do not know the compatibility state of dependencies. Best, Arthur On Wed, Jun 1, 2016 at 2:59 PM, Bence

Re: Why do we need SQLite in Airflow?

+1, but it feels like just piling on. One thing we could consider is which part we would like to fix. - If it is the seriousness/production ready db, but that is still a local db/client, we could try something like firebird. Relatively small footprint and can do multithreading, it is supported

Re: Apache Airflow (Incubating) : May Board Report Draft (comments welcome)

+1 as well. On Fri, Apr 29, 2016 at 2:59 PM, Bolke de Bruin wrote: > Really that many PRs? That’s crazy. I think I need some time off :) > > Do you want to mention any migration issues? ie. not allowing us to use > the cool stuff :)? > > +1 LGTM > > Bolke > > > Op 29 apr.