Hey Luke, Who or what makes the decision to skip processing that task?
Rgds, Gerard On Wed, Nov 9, 2016 at 2:39 PM, Maycock, Luke < [email protected]> wrote: > Hi Gerard, > > > Thank you for your quick response. > > > I am not trying to implement this for a specific operator but rather > trying to add it as a feature for any task in any DAG. > > > Given that the skipped states propagate where all directly upstream tasks > are skipped, I don't think this is the state we want to use. For the > functionality I'm looking for, I think I'll need to introduce a new status, > maybe 'disabled'. > > > Again, thanks for your response. > > > Cheers, > Luke Maycock > OLIVER WYMAN > [email protected]<mailto:luke. > [email protected]> > www.oliverwyman.com<http://www.oliverwyman.com/> > > > > ________________________________ > From: Gerard Toonstra <[email protected]> > Sent: 08 November 2016 18:19 > To: [email protected] > Subject: Re: Skip task > > Also in 1.7.1.3, there's the ShortCircuitOperator, which can give you an > example. > > https://github.com/apache/incubator-airflow/blob/1.7.1. > 3/airflow/operators/python_operator.py > > You'd have to modify this to your needs, but the way it works is that if > the condition evaluates to True, none of the > downstream tasks are actually executed, they'd be skipped. The reason for > putting them into SKIPPED state is that > the DAG final result would still be SUCCESS and not failed. > > You could copy the operator from there and don't do the full "for loop", > only pick the tasks immediately downstream > from this operator and skip that. Or... if you need to skip additional > tasks downstream, add a parameter "num_tasks" > that decide on a halting condition for the for loop. > > I believe that should work. I didn't try that here, but you can test that > and see what it does for you. > > > If you want this as a UI capability... for example have a human operator > decide on skipping this yes or not, then > maybe the best way forward would be some kind of highly custom plugin with > its own view. In the end, you'd basically > do the same action in the backend, whether the python cond evaluates to > True or the button is clicked. > > In the plugin case though, you'd have to keep the UI and the structure of > the DAG in sync and aligned, otherwise > it'd become a mess.... Airflow wasn't really developed for workflow/human > interaction, but in workflows where only > automated processes are involved. That doesn't mean that you can't do > anything like that, but it may be costly resource > wise to get this done. For example, on the basis of the BranchOperator, you > could call an external API to verify if a decision > was taken on a case, then follow branch A or B if the decision is there or > put the state back into UP_FOR_RETRY. > At the moment though, there's no programmatic way to reschedule that task > to some minutes or hours into the future before > it's looked at again, unless you really dive into airflow, scheduling > semantics (@once vs. other schedules) and how > the scheduler works. > > Rgds, > > Gerard > > > > > On Tue, Nov 8, 2016 at 5:30 PM, Maycock, Luke < > [email protected]> wrote: > > > Hi All, > > > > > > I am using Airflow 1.7.1.3 and have a particular requirement, which I > > don't think is currently supported by Airflow but just wanted to check in > > case I was missing something. > > > > > > I occasionally wish to skip a particular task in a given DAG run such > that > > the task does not run for that DAG run. Is this functionality available > in > > Airflow? > > > > > > I am aware of the BranchPythonOperator (https://airflow.incubator. > > apache.org/concepts.html#branching) but I don't think believe this is > > exactly what I am looking for. > > > > > > I am thinking that a button in the UI alongside the 'Mark Success' and > > 'Run' buttons would be appropriate. > > > > > > If the functionality does not exist, does anyone have any suggestions on > > ways to implement this? > > > > > > Cheers, > > Luke Maycock > > OLIVER WYMAN > > [email protected]<mailto:luke. > > [email protected]> > > www.oliverwyman.com<http://www.oliverwyman.com/> > > > > > > ________________________________ > > This e-mail and any attachments may be confidential or legally > privileged. > > If you received this message in error or are not the intended recipient, > > you should destroy the e-mail message and any attachments or copies, and > > you are prohibited from retaining, distributing, disclosing or using any > > information contained herein. Please inform us of the erroneous delivery > by > > return e-mail. Thank you for your cooperation. > > > > ________________________________ > This e-mail and any attachments may be confidential or legally privileged. > If you received this message in error or are not the intended recipient, > you should destroy the e-mail message and any attachments or copies, and > you are prohibited from retaining, distributing, disclosing or using any > information contained herein. Please inform us of the erroneous delivery by > return e-mail. Thank you for your cooperation. >
