Without this change does on_kill ever get triggered? It seems like this change is desired behaviour.
As per the first comment https://issues.apache.org/jira/browse/AIRFLOW-1623?focusedCommentId=16171819&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-16171819 <https://issues.apache.org/jira/browse/AIRFLOW-1623?focusedCommentId=16171819&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-16171819> I'm not sure this is the right fix. It also seems like this would end up running the on_kill in a different process to the rest of the operator. I wonder if somewhere a signal handler is missing somewhere in one of the `run --local` or `run --raw`. I tried to follow all the paths through from ui to sig handlers but got stuck in a tiwsty maze of classes. (and was attempting to do it just from reading the code)? > On 8 Jan 2018, at 13:15, Driesprong, Fokko <[email protected]> wrote: > > Yes, for Spark this should work. Depending on the operator and the > implementation: > https://github.com/apache/incubator-airflow/blob/3e6babe8ed8f8f281b67aa3f4e03bf3cfc1bcbaa/airflow/contrib/hooks/spark_submit_hook.py#L412-L428 > > However this is a big change in behaviour. I'm curious about the opinion of > others. > > Cheers, > Fokko > > > 2018-01-08 14:12 GMT+01:00 Milan van der Meer < > [email protected]>: > >> Any help? :) >> >> On Thu, Dec 14, 2017 at 8:12 PM, Milan van der Meer < >> [email protected]> wrote: >> >>> I recently openend the following PR: https://github.com/apache/ >>> incubator-airflow/pull/2877 >>> >>> The problem is that on_kill is not called for operators when you clear a >>> task from the UI. >>> Thats problematic when working with ex. spark clusters as the jobs on the >>> cluster need to be killed. >>> >>> The issue is in the core code of Airflow and Im not familiar enough with >>> the inner workings there. So I could use some directions on this one from >>> people who are familiar. >>> >>> For more info, check out the PR. >>> >>> Kind regards, >>> Milan >>> >> >> >> >> -- >> >> *Milan van der Meer* >> >> *Real**Impact* Analytics *| *Big Data Consultant >> www.realimpactanalytics.com >> >> *BE *+32 498 45 96 22 <0032498459622>* | Skype *milan.vandermeer.ria >>
