Hey, I agree, I always thought this would be handled through a BaseOperator flag `only_run_latest=True`. There are some potentially confusing / incompatible scenarios like: * can't have both depend_on_past and only_run_latest (obviously...) * can't have any downstream tasks of only_run_latest have depend_on_past * same 2 statements also apply to "wait_for_downstream" * more?
Max On Fri, Aug 10, 2018 at 12:43 AM Kevin Campbell <[email protected]> wrote: > Hi everyone, > > We had some issues with the ShortCircuitOperator, so had to come up with an > alternative implementation. > > I've written up the details and the workaround in this blog posting: > > > > https://blog.diffractive.io/2018/08/07/replacement-shortcircuitoperator-for-airflow/ > > Seems that to fix this in airflow properly, it needs some design changes. I > think the sanest solution would be to have the scheduler handle skip > operations, rather than implement this inside an operator. That would mean > either having the scheduler read xcom data, or elevating the status of the > python member to a first class attribute on a task. > > Anyway, hoping this posting might at least save some users some headaches > if they want the ShortCircuitOperator to behave better. > > Comments and feedback appreciated. > > Regards, > Kevin >
