Re: scheduler questions

<mailto:luke.mayc...@affiliate.oliverwyman.com> www.oliverwyman.com<http://www.oliverwyman.com/> From: siddharth anand <san...@apache.org> Sent: 13 October 2016 18:11 To: dev@airflow.incubator.apache.org Subject: Re: scheduler questions B

Re: scheduler questions

you rock, Sid! thanks for taking your time explaining it for me On Thu, Oct 13, 2016 at 6:10 PM, siddharth anand wrote: > I can't see an image. > > We run most of our dags with depends_on_past=True. > > If you want to chain your dag runs, such as not starting the first task

Re: scheduler questions

I can't see an image. We run most of our dags with depends_on_past=True. If you want to chain your dag runs, such as not starting the first task of your dag run start until the last task of your previous dag runs completes, you can use an external task sensor. The external task sensor would be

Re: scheduler questions

This is not what I see actually. I posted below my test DAG and a screenshot. It does create DAGRuns on subsequent runs - I modeled that scenario by commenting one bash command and uncommenting another one with Exit 1. it does not create Task Instances on subsequent failed DAGs but it does

Re: scheduler questions

If you use depends_on_past=True, it won't proceed to the next DAG Run if the previous DAG Run failed. If Day 2 fails, Day 3 won't run. -s On Thu, Oct 13, 2016 at 10:34 AM, siddharth anand wrote: > Yes! It does work with Depends_on_past=True. > -s > > On Thu, Oct 13, 2016 at

Re: scheduler questions

Yes! It does work with Depends_on_past=True. -s On Thu, Oct 13, 2016 at 10:28 AM, Boris Tyukin wrote: > thanks so much, Sid! just a follow up question on "Only_Run_Latest" - will > it work with depend_on_past = True? or it will assume that DAG is used > False? > > On Thu,

Re: scheduler questions

so for my second scenario, I think i would still need to run missing days jobs one by one (by clearing the failed ones) and I understand this is recommended approach as I figured from Maxime's video. But sometimes it is more efficient to combine all missing day runs in one so I would be using a

Re: scheduler questions

Hi Boris, To answer the first question, the backfill command has a flag to mark jobs as successful without running them. Take care to align the start and end times precisely as needed. As an example, for a job that runs daily at 7am: airflow backfill -s 2016-10-07T07 -e 2016-10-10T07

scheduler questions

Hello all and thanks for such an amazing project! I have been evaluating Airflow and spent a few days reading about it and playing with it and I have a few questions that I struggle to understand. Let's say I have a simple DAG that runs once a day and it is doing a full reload of tables from the