Hi all, After discussion at the NY Meetup this week I've been pondering how Airflow could support custom schedules with very little change to core Airflow logic and keeping backwards compatibility.
As I understand the common way to support custom schedules is through a BranchOperator. You provide logic that on a good date executes the "run" branch and on another date runs the "don't run" branch which usually is a dummy operator. There are 2 problems associated with it which would be useful to me (and I think the rest of the community) to solve: 1. depends_on_past does not play well with branching, because the "run" branch tasks get marked as "skipped" 2. Template variables like "prev_ds" and "next_ds" represent the underlying schedule and not the actual schedule you are working on I therefore propose a "schedule_filter_callback", a function which you provide at DAG creation time that takes in some arguments (execution date, timezone, DAG?), and returns a Truthy or Falsy result based on if this is a good date to execute on. If schedule_filter_callback is None then the current schedule logic is applied. I appreciate this is a fairly significant proposal, but it seems like because it would just be 1 extra argument on the DAG and make no change to the default behavior it doesn't quite rise to the level of AIP? Sorry if this has already been discussed before. Regards, Damian =============================================================================== Please access the attached hyperlink for an important electronic communications disclaimer: http://www.credit-suisse.com/legal/en/disclaimer_email_ib.html ===============================================================================
