Yes, for Spark this should work. Depending on the operator and the implementation: https://github.com/apache/incubator-airflow/blob/3e6babe8ed8f8f281b67aa3f4e03bf3cfc1bcbaa/airflow/contrib/hooks/spark_submit_hook.py#L412-L428
However this is a big change in behaviour. I'm curious about the opinion of others. Cheers, Fokko 2018-01-08 14:12 GMT+01:00 Milan van der Meer < [email protected]>: > Any help? :) > > On Thu, Dec 14, 2017 at 8:12 PM, Milan van der Meer < > [email protected]> wrote: > > > I recently openend the following PR: https://github.com/apache/ > > incubator-airflow/pull/2877 > > > > The problem is that on_kill is not called for operators when you clear a > > task from the UI. > > Thats problematic when working with ex. spark clusters as the jobs on the > > cluster need to be killed. > > > > The issue is in the core code of Airflow and Im not familiar enough with > > the inner workings there. So I could use some directions on this one from > > people who are familiar. > > > > For more info, check out the PR. > > > > Kind regards, > > Milan > > > > > > -- > > *Milan van der Meer* > > *Real**Impact* Analytics *| *Big Data Consultant > www.realimpactanalytics.com > > *BE *+32 498 45 96 22 <0032498459622>* | Skype *milan.vandermeer.ria >
