Quick note to say that all this is not ideal or the way it should be, I'm just describing how it currently works and how it is documented. I'm very open to ideas to make all this more intuitive or making the documentation clearer.
A random idea would be to have a DagRun boolean attribute "disregard_depend_on_past" for this DagRun that could be altered in the UI. @aoen is coming up with a page that should bring clarity around "why isn't this task_instance running?" which can have in-place tooltips around how to get things going. On Tue, Apr 26, 2016 at 8:08 AM, Maxime Beauchemin < [email protected]> wrote: > >>> "The clear fix seems to be to have depends_on_past check the last TI > that > ran, regardless of whether it ran `schedule_interval` ago. That's in line > with the intent of the flag. I will submit a fix." > > I don't think so. This would lead to skipping runs, which depends_on_past > is used as a guarantee to run every TI, sequentially. > > Absolute scheduling (cron expressions) is much better than relative > scheduling (origin + interval). Though it's easy to make relative > scheduling behave in an absolute way. You just have to use a rounded > start_date to your schedule_interval, and not move things around. Dynamic > start_dates have always been a problem and should probably not be > supported, though there's no way for us to tell. > > Changing the schedule interval or the "origin time" is a bit tricky and > should be documented. > > If people use depend_on_past=True and change the origin or the interval, > they basically redefine what "past" actually means and will require to > "mark success" or defining a new "start_date" as a way to say "please > disregard depend_on_past for this date" > > For those who haven't fully digested "What's the deal with start_dates", > please take the time to read it: > http://pythonhosted.org/airflow/faq.html > > Also see this part of the docs: > > > > Max > > On Mon, Apr 25, 2016 at 1:14 PM, Jeremiah Lowin <[email protected]> wrote: > >> Bolke, Sid and I had a brief conversation to discuss some of the >> implications of https://github.com/airbnb/airflow/issues/1427 >> >> There are two large points that need to be addressed: >> >> 1. this particular issue arises because of an alignment issue between >> start_date and schedule_interval. This can only happen with cron-based >> schedule_intervals that describe absolute points in time (like “1am”) as >> opposed to time deltas (like “every hour”). Ironically, I once reported >> this same issue myself (#959). In the past (and in the docs) we have >> simply >> said that users must make sure the two params agree. We discussed the >> possibility of a DAG validation method to raise an error if the start_date >> and schedule_interval don’t align, but Bolke made the point (and I agreed) >> that in these cases, start_date is sort of like telling the scheduler to >> “start paying attention” as opposed to “this is my first execution date”. >> In #1427, the scheduler was being asked to start paying attention on >> 4/24/16 00:00:00 but not to do anything until 4/24/16 01:10:00. However, >> it >> was scheduling a first run at midnight and a second run at 1:10. >> >> Regardless of whether we choose to validate/warn/error, Bolke is going to >> change the scheduling logic so that the cron-based interval takes >> precedence over a start date. Specifically, the first date on or after the >> start_date that complies with the schedule_interval becomes the first >> execution date. >> >> 2. Issue #1 led to a second issue: depends_on_past checks for a successful >> TI at `execution_date - schedule_interval`. This is fragile, since it is >> very possible for the previous TI to have run at any time in the past, not >> just one schedule_interval ago. This can happen easily with ad-hoc DAG >> runs, and also if a DAG was paused for a while. Less commonly, it happens >> with the situation described in point #1, where the first scheduled run is >> off-schedule (the midnight run followed by the daily 1:10am runs). >> >> The clear fix seems to be to have depends_on_past check the last TI that >> ran, regardless of whether it ran `schedule_interval` ago. That's in line >> with the intent of the flag. I will submit a fix. >> >> -J >> > >
