DAG design for interacting with APIs

I need help designing a DAG High level problem: I need a way to launch tasks through an API and manage their state, when they fail I need the ability to automatically retry. What's involved: We use Singularity (https://github.com/HubSpot/Singularity) to launch tasks on Mesos which can be

Re: Airflow Meetup @ Paypal (San Jose)

Jayesh, Looks good. No need to vote. Just publish a new event with details on the meet-up page: https://www.meetup.com/Bay-Area-Apache-Airflow-Incubating-Meetup/ Please add a short abstract as well for the talks and find a 3rd speaker. Please be sure to record the meet-up so that we can publish

Re: Airflow 1.8.0 BETA 5

Rc 1 isn't out yet, differences will be small. Sent from my iPhone > On 31 Jan 2017, at 22:00, Chris Riccomini wrote: > > I'm still on b5. Should I upgrade to rc1, or is it still in progress? > >> On Tue, Jan 31, 2017 at 12:26 PM, Alex Van Boxel

Re: Airflow 1.8.0 BETA 5

I'm still on b5. Should I upgrade to rc1, or is it still in progress? On Tue, Jan 31, 2017 at 12:26 PM, Alex Van Boxel wrote: > I identified my root cause: it's was a problem at my site with a faulty > rebase. So all green. > > On Tue, Jan 31, 2017 at 8:52 PM Alex Van Boxel

Re: Airflow 1.8.0 BETA 5

I identified my root cause: it's was a problem at my site with a faulty rebase. So all green. On Tue, Jan 31, 2017 at 8:52 PM Alex Van Boxel wrote: > So bumped to RC1 and this seems fine. I don't get it. > > So it's a go. Sorry for the noise, but better safe then sorry. I also

Re: Airflow 1.8.0 BETA 5

So bumped to RC1 and this seems fine. I don't get it. So it's a go. Sorry for the noise, but better safe then sorry. I also found that the scheduler logs setting are not equal from the normal logging so that's also a go. You get green light from me. On Tue, Jan 31, 2017 at 8:32 PM Bolke de

Re: Airflow Meetup @ Paypal (San Jose)

Cool! On Tue, Jan 31, 2017 at 10:20 AM, Boris Tyukin wrote: > yay awesome! thanks *Jayesh!* > > On Tue, Jan 31, 2017 at 12:19 PM, Jayesh Senjaliya > wrote: > > > HI All, > > > > √ I have approval from Paypal to host Airflow meetup. How about March >

Re: Airflow 1.8.0 BETA 5

And the scheduler doesn't log anything in beta 4? Sent from my iPhone > On 31 Jan 2017, at 19:26, Alex Van Boxel wrote: > > I see it in the scheduler and in the UI. Currently trying to do a new > upgrade. > >> On Tue, Jan 31, 2017 at 8:00 PM Bolke de Bruin

Re: Airflow 1.8.0 BETA 5

I see it in the scheduler and in the UI. Currently trying to do a new upgrade. On Tue, Jan 31, 2017 at 8:00 PM Bolke de Bruin wrote: > Please note I will be holding off on the RC, before we understand this > issue better. > > Bolke > > > On 31 Jan 2017, at 18:06, Bolke de

Re: Airflow 1.8.0 BETA 5

Please note I will be holding off on the RC, before we understand this issue better. Bolke > On 31 Jan 2017, at 18:06, Bolke de Bruin wrote: > > Hey Alex, > > Could it actually be that Airflow is doing the right thing? Earlier it was > swallowing the errors. Where do you

Re: nvd3 is an external resource

Can you please test https://github.com/apache/incubator-airflow/pull/2035 and +1 if it works for you. Thanks Bolke > On 30 Jan 2017, at 12:04, Сёмочкин Максим Викторович wrote: > > I think I found a way to prevent the generation of unnecessary headers in the > html code. >

Re: Airflow Meetup @ Paypal (San Jose)

yay awesome! thanks *Jayesh!* On Tue, Jan 31, 2017 at 12:19 PM, Jayesh Senjaliya wrote: > HI All, > > √ I have approval from Paypal to host Airflow meetup. How about March 14th > ? Please vote. > > √ we will have food and drinks. > Please let me know if anybody has any

Re: Airflow 1.8.0 BETA 5

Hey Alex, Could it actually be that Airflow is doing the right thing? Earlier it was swallowing the errors. Where do you see the errors? In the scheduler logs, UI, processor logs? - Bolke > On 31 Jan 2017, at 16:25, Alex Van Boxel wrote: > > I'll try to identify the core

Re: Airflow 1.8.0 BETA 5

Data point: b5 is working fine for me so far. On Tue, Jan 31, 2017 at 8:25 AM, Alex Van Boxel wrote: > I'll try to identify the core problem > > On Tue, Jan 31, 2017, 16:43 Bolke de Bruin wrote: > > > Hey Alex > > > > Can you provide some info on the

Re: Airflow 1.8.0 BETA 5

I'll try to identify the core problem On Tue, Jan 31, 2017, 16:43 Bolke de Bruin wrote: > Hey Alex > > Can you provide some info on the scheduler paths thing. I don't have/see > that issue. Do you mean cli paths or by cfg? Jira would be nice in any case. > > I don't think the

Re: Airflow 1.8.0 BETA 5

Hey Alex Can you provide some info on the scheduler paths thing. I don't have/see that issue. Do you mean cli paths or by cfg? Jira would be nice in any case. I don't think the dag processor respects cli parameters. Bolke Sent from my iPhone > On 31 Jan 2017, at 15:10, Alex Van Boxel

Preparing for 1.8.0 Release Candidate (1) - Reviews needed

Hey All, We are very close to having a release candidate for 1.8! I would like to get some small fixes in to make sure we get a smooth ride and hopefully graduate the RC quickly to release: * https://github.com/apache/incubator-airflow/pull/2039 Fixes Python 3 compatibility and enables all