Re: Article - Get started developing workflows with Apache Airflow

Very nice. I noticed a bug in one line: pip install pip install airflow==1.8.0 I see you've used the plugin class to add operators, so they appear in the airflow.operators namespace. I'm wondering about what other people are doing there and what the best way is to add custom operators to

Re: Article - Get started developing workflows with Apache Airflow

Yes, we do actually use virtualenv in our Airflow production environment.We also use Docker containers, so we don't strictly need it, but it makes it easy for us to have a similar setup in production and on developers' machines.On dev machines, we run multiple Python projects each hosted in its

Podling Report Reminder - April 2017

Dear podling, This email was sent by an automated system on behalf of the Apache Incubator PMC. It is an initial reminder to give you plenty of time to prepare your quarterly board report. The board meeting is scheduled for Wed, 19 April 2017, 10:30 am PDT. The report for your podling will form

Re: Cleanup

Edgardo, This is a great question and something that requires functionality to address. As Airflow starts getting used for bigger workloads, we need a way to clean up defunct resources. - How do we delete a dag and its related resources? - Until the recent release, the way that I stopped

Re: Cleanup

To add to Siddharth's pretty extensive list (in particular, the "delete a DAG from the code that makes up the dag bag folder, but now it shows up with a ! icon and you have to manually set it to is_active = f" issue that I didn't see in 1.8.0-RC4 but started seeing in 1.8.0-RC5 that became 1.8.0)

Problem using SSL w/ self-signed cert?

All, I've generated a self signed certificate to use with airflow using the following command: openssl req -x509 -newkey rsa:4096 -sha256 -nodes -keyout key.pem -out cert.pem -days 365 However when trying to access the site from the local machine via: curl -k https://localhost curl hangs and

Re: 1.8.1 release

https://issues.apache.org/jira/browse/AIRFLOW-1033 Is also 1.8 and not 1.8.0 blocker ? Thanks. -- Ruslan Dautkhanov On Mon, Apr 3, 2017 at 3:20 PM, Bolke de Bruin wrote: > Hi Chris, > > Please add: > > https://issues.apache.org/jira/browse/AIRFLOW-1011 < >

Re: Podling Report Reminder - April 2017

I will do it. I will work on it today as it seems like today is the last day to do this. On Wed, Apr 5, 2017 at 10:29 AM, Chris Riccomini wrote: > Can someone please volunteer to update? Should take ~5 minutes. > > On Tue, Apr 4, 2017 at 8:16 PM,

Re: Podling Report Reminder - April 2017

Can someone please volunteer to update? Should take ~5 minutes. On Tue, Apr 4, 2017 at 8:16 PM, wrote: > Dear podling, > > This email was sent by an automated system on behalf of the Apache > Incubator PMC. It is an initial reminder to give you plenty of time to > prepare

Re: Podling Report Reminder - April 2017

:-) If you can please include our intention to graduate after the 1.8.1 release (no more reports ;-). B. > On 5 Apr 2017, at 19:32, Gurer Kiratli > wrote: > > I will do it. I will work on it today as it seems like today is the last > day to do this. > > On

Re: Podling Report Reminder - April 2017

Sure thing! > On Apr 5, 2017, at 10:33 AM, Bolke de Bruin wrote: > > :-) > > If you can please include our intention to graduate after the 1.8.1 release > (no more reports ;-). > > B. > >> On 5 Apr 2017, at 19:32, Gurer Kiratli >>

PTAL: Airflow 2017 April Podling Report

Hi folks, Here is the draft of the podling report. Please take a look and comment. If it looks good one of the committers have to post this on this on the wiki today! Cheers, Gurer >> Airflow Airflow is a workflow automation and scheduling system that can be used to author and manage data

Re: Podling Report Reminder - April 2017

Awesome, thanks Gurer! On Wed, Apr 5, 2017 at 10:34 AM, Gurer Kiratli < gurer.kira...@airbnb.com.invalid> wrote: > Sure thing! > > > On Apr 5, 2017, at 10:33 AM, Bolke de Bruin wrote: > > > > :-) > > > > If you can please include our intention to graduate after the 1.8.1 >

Re: PTAL: Airflow 2017 April Podling Report

Lgtm2 :-) Sent from my iPhone > On 5 Apr 2017, at 21:38, Chris Riccomini wrote: > > LGTM! Thanks! > > On Wed, Apr 5, 2017 at 11:57 AM, Gurer Kiratli < > gurer.kira...@airbnb.com.invalid> wrote: > >> Hi folks, >> >> Here is the draft of the podling report. Please take

Re: Problem using SSL w/ self-signed cert?

Please disregard this question, it was apparently my load balancer generating the errors and curl was taking some time to come back, however it does eventually come back. -N nik.hodgkin...@collectivehealth.com On Wed, Apr 5, 2017 at 6:44 PM, Nicholas Hodgkinson

Re: Podling Report Reminder - April 2017

On 5 April 2017 at 10:33, Bolke de Bruin wrote: > If you can please include our intention to graduate after the 1.8.1 release > (no more reports ;-). Did I miss this discussion on the mailing list? If so, can you point me to it? If it was discussed there; there may be some

Memory Issues with Airflow Subdag

Hey Everyone, We recently added a subdag with over 200 tasks on it and any time airflow is running the cpu and memory usage spikes taking down the server. I am putting the subdag in a factory outside of the dags folder and importing it into a dag. We are using the Celery Executor, and running the

Re: PTAL: Airflow 2017 April Podling Report

I added the following to the report, as it seemed like it was a required question : How would you assess the podling's maturity?Please feel free to add your own commentary. [ ] Initial setup [ ] Working towards first release [ ] Community building [X] Nearing graduation [ ] Other:The Airflow

Airflow 1.8 crash on python 3.4

Hi all, Hope you're well. I implement an airflow dag and I have an issue which I don't understand. I work on a docker stack, with : 1 redis, 1 scheduler and 1 worker container on the same instance. On scheduler and worker I install* python 3.4* (airflow 1.8) and distributed my task with