Re: Medium series: Airflow for Google Cloud

Looks like you don't have an account.. once you create one.. let me know and I will grant you admin perms on the wiki. -s On Fri, Jan 20, 2017 at 6:08 PM, siddharth anand wrote: > I've added it to https://cwiki.apache.org/confluence/display/AIRFLOW/ > Airflow+Links > > Feel

Re: Medium series: Airflow for Google Cloud

I've added it to https://cwiki.apache.org/confluence/display/AIRFLOW/Airflow+Links Feel free to add future posts to this page. You should have access. -s On Fri, Jan 20, 2017 at 3:23 PM, Alex Van Boxel wrote: > Hey all, > > now that 1.8 is nearing release. I finally started

Article: The Rise of the Data Engineer

Hey I just published an article about the "Data Engineer" role in modern organizations and thought it could be of interest to this community. https://medium.com/@maximebeauchemin/the-rise-of-the-data-engineer-91be18f1e603#.5rkm4htnf Max

Re: Airflow 1.8.0 BETA 1

I completely understand what your trying to achieve, but I'm just not sure your get that result by using a 1.7 scheduler with a 1.8 worker, exactly because the contract is so simple and the worker itself doesn’t do too much (although the dependency engine has changed as well, so the “why isn’t

Medium series: Airflow for Google Cloud

Hey all, now that 1.8 is nearing release. I finally started writing about Airflow. As it's me writing, I'll be focussing on the Google Cloud integration. Today's post is about BigQuery https://medium.com/google-cloud/airflow-for-google-cloud-part-1-d7da9a048aa4#.qe6f0gldf Next one will be about

Re: Airflow 1.8.0 BETA 1

The benefit is really just to limit the scope of the errors as we proceed cautiously, progressively with more confidence. As in we upgrade one small low SLA queue first (set of workers), find some worker-related bugs, web server bugs, fix them. Rinse and repeat until all workers are on 1.8.0. Then

Re: Airflow 1.8.0 BETA 1

Hi Max, Interesting idea. I agree with your assumption that the contract between the scheduler and the worker is pretty simple and it may work for upgrades where this contract hasn’t been altered. However, between plain 1.7.1 and 1.8.0 this contract has significantly changed. The handover to

Re: Experiences with 1.8.0

I'd be happy to lend a hand fixing these issues and hopefully some others are too. Do you mind creating jiras for these since you have the full context? I have created a JIRA for (1) and have assigned it to myself: https://issues.apache.org/jira/browse/AIRFLOW-780 On Fri, Jan 20, 2017 at 1:01 AM,

Re: Airflow 1.8.0 BETA 1

Hi all, I need some input around this progressive upgrade idea I had recently. At Airbnb we have many queues of workers, and I was entertaining the idea of rolling out 1.8.0beta in production on a per worker or per-queue basis to minimize the risks around upgrading. This of course assumes that

Re: Airflow Meetup @ Paypal (San Jose)

I think if we hold it in the evening, there is no requirement to buy a ticket to come to the meetup. Let me verify. On Fri, Jan 20, 2017 at 12:45 PM, Jayesh Senjaliya wrote: > Hi Russell, > > Sure, Strata will have its own flavor of visitors, but the tickets are > kind of

Re: Airflow Meetup @ Paypal (San Jose)

Hi Russell, Sure, Strata will have its own flavor of visitors, but the tickets are kind of expensive too for everybody to join. I agree on turnouts though, so we can try for Strata first and fallback to regular meetup in March end or even April if we dont get space in Strata. or we can just do

Re: Airflow Meetup @ Paypal (San Jose)

As I mentioned in the other thread, I am available to speak on Predictive Analytics with Airflow and PySpark. Mid march has been suggested. What about the evening of Tuesday, 3/14 - the first day of sessions at Strata? We could promote the meetup with the conference, get it listed as an evening

Re: Airflow Meetup in NYC @ Blue Apron

Cool, would you have remote joining setup (hangout?, adobe?) or recording for this for the folks not in NYC ? Thanks for hosting ! On Fri, Jan 20, 2017 at 10:37 AM, Joseph Napolitano < joseph.napolit...@blueapron.com.invalid> wrote: > Hi all! > > I want to officially announce a Meetup for

Re: Airflow 1.8.0 BETA 2

Installed in dev. Prod will go on Monday. Will keep you posted. On Fri, Jan 20, 2017 at 9:35 AM, Bolke de Bruin wrote: > Yes > > Sent from my iPhone > > > On 20 Jan 2017, at 18:20, Boris Tyukin wrote: > > > > just to make sure this is the latest one,

Airflow Meetup in NYC @ Blue Apron

Hi all! I want to officially announce a Meetup for Airflow in NYC! I'm looking forward to meeting other community members to share knowledge and network. We may create an official Meetup page, but in the meantime please signup here: https://docs.google.com/spreadsheets/d/1WmfgZeExSVdLf-

Re: NYC Meetup?

Hi All, I wanted to bump this thread again. I sent out another email about a meetup in NYC, so look for that one. It took a long time to get approved over the holidays, so I hope we can still generate interest in a short time. Cheers! On Thu, Dec 29, 2016 at 3:01 PM, Joseph Napolitano <

Re: Airflow 1.8.0 BETA 2

Yes Sent from my iPhone > On 20 Jan 2017, at 18:20, Boris Tyukin wrote: > > just to make sure this is the latest one, right? > https://dist.apache.org/repos/dist/dev/incubator/airflow/airflow-1.8.0b2+apache.incubating.tar.gz > >> On Fri, Jan 20, 2017 at 10:57 AM, Bolke

Re: New book covers Airflow with PySpark: Agile Data Science 2.0 (O'Reilly, 2017) AND Airflow Meetup?

Let me email about this with its own email subject. On Thu, Jan 19, 2017 at 10:54 PM Jayesh Senjaliya wrote: > Hi Siddharth, > > I am Jayesh from Paypal, and at last meetup we briefly talked about > hosting next one and I offered to host next Airflow meetup at Paypal >

Re: Airflow 1.8.0 BETA 2

just to make sure this is the latest one, right? https://dist.apache.org/repos/dist/dev/incubator/airflow/airflow-1.8.0b2+apache.incubating.tar.gz On Fri, Jan 20, 2017 at 10:57 AM, Bolke de Bruin wrote: > Hi All, > > I have made the SECOND beta of Airflow 1.8.0 available at:

Airflow 1.8.0 BETA 2

Hi All, I have made the SECOND beta of Airflow 1.8.0 available at: https://dist.apache.org/repos/dist/dev/incubator/airflow/ , public keys are available at https://dist.apache.org/repos/dist/release/incubator/airflow/

Re: How to learn more about deprecation warnings?

Hi Laura, The error is raised if an unused argument is passed to BaseOperator -- basically if there is anything in either args or kwargs. The original issue was that in a number of cases arguments were misspelled or misused by Operator subclasses and instead of raising an error, they were just

Experiences with 1.8.0 (updated)

— continued accidentally pressed send — This is to report back on some of the (early) experiences we have with Airflow 1.8.0 (beta 1 at the moment): 1. The UI does not show faulty DAG, leading to confusion for developers. When a faulty dag is placed in the dags folder the UI would report a

Experiences with 1.8.0

This is to report back on some of the (early) experiences we have with Airflow 1.8.0 (beta 1 at the moment): 1. The UI does not show faulty DAG, leading to confusion for developers. When a faulty dag is placed in the dags folder the UI would report a parsing error. Now it doesn’t due to the

Re: Airflow 1.8.0 BETA 1

1. Always do backups 2. Your airflow.cfg will work, but you might want to adjust some settings that are new 3. Pip install https://dist.apache.org/repos/dist/dev/incubator/airflow/airflow-1.8.0b1+apache.incubating.tar.gz should work. > On 19 Jan 2017, at 23:25, Boris Tyukin