Thanks Joy - that solution works for me. However I found then that the state of the instance is lost - apparently the operator constructor is called again. For example the BashOperator stores a reference to the process in self.sp: https://github.com/apache/incubator-airflow/blob/master/airflow/operators/bash_operator.py#L89 And then calling on_kill will raise an AttributeError - is this as expected?
On Wed, Sep 13, 2017 at 2:50 PM, Joy Gao <[email protected]> wrote: > Hi Richard, > > Can confirm that the operator's on_kill() is not triggered via `clear`, > only job.on_kill() is triggered, but it doesn't propagate to the > operator/task. From what I can tell, currently the only way to trigger > on_kill() is via manually sending a SIGTERM to the process while the task > is running (code ref > <https://github.com/apache/incubator-airflow/blob/a8d486de337779e6319b90215fd8ea7faa5f5913/airflow/models.py#L1436-L1441>), > or if the task actually times out (code ref > <https://github.com/apache/incubator-airflow/blob/a8d486de337779e6319b90215fd8ea7faa5f5913/airflow/models.py#L1458> > ). > > I assume that the intended behavior is to trigger operator's on_kill() > method whenever a task is cleared. If so, I'd propose patching > theLocalTaskJob's on_kill() method as below in a PR: > > def on_kill(self): > * self.task_instance.task.on_kill()* > self.task_runner.terminate() > self.task_runner.on_finish() > > Hope this helps! > > On Wed, Sep 13, 2017 at 8:37 AM, Richard Baron Penman < > [email protected]> wrote: > >> Thanks Ash. >> In my logs I see "Job shut down externally", so that BaseJob.kill() >> method is certainly executed. >> >> My understanding is this then calls self.on_kill(): >> https://github.com/apache/incubator-airflow/blob/ >> b2e1753f5b74ad1b6e0889f7b784ce69623c95ce/airflow/jobs.py#L2512 >> >> which terminates the process: >> https://github.com/apache/incubator-airflow/blob/ >> b2e1753f5b74ad1b6e0889f7b784ce69623c95ce/airflow/task_ >> runner/bash_task_runner.py#L34 >> >> How is the operators on_kill() method then triggered? According to my >> logging it is not executed. >> >> >> On Fri, Sep 8, 2017 at 4:49 AM, Ash Berlin-Taylor >> <[email protected]> wrote: >> > That sets the task instance state to SHUTDOWN, which is then picked up >> by the task sentinel here https://github.com/apache/ >> incubator-airflow/blob/b2e1753f5b74ad1b6e0889f7b784ce >> 69623c95ce/airflow/jobs.py#L164-L165 <https://github.com/apache/ >> incubator-airflow/blob/b2e1753f5b74ad1b6e0889f7b784ce >> 69623c95ce/airflow/jobs.py#L164-L165> which triggers kill, so it is meant >> to work. >> > >> > -ash >> > >> >> On 7 Sep 2017, at 20:00, Richard Baron Penman <[email protected]> >> wrote: >> >> >> >> Is this known to work? >> >> I tried clearing and the on_kill() method wasn't triggered. >> >> >> >> As far as I can tell the clearing is performed by the >> >> clear_task_instances() function, which changes the task status but does >> not >> >> kill the process: >> >> https://github.com/apache/incubator-airflow/blob/ >> b2e1753f5b74ad1b6e0889f7b784ce69623c95ce/airflow/models.py#L120 >> >> >> >> >> >> >> >> On Thu, Sep 7, 2017 at 4:46 AM, Maxime Beauchemin < >> >> [email protected]> wrote: >> >> >> >>> "Clearing" a running task instance either through the UI or through >> the CLI >> >>> should trigger on_kill. >> >>> >> >>> Max >> >>> >> >>> On Wed, Sep 6, 2017 at 11:45 AM, Richard Baron Penman < >> [email protected] >> >>>> >> >>> wrote: >> >>> >> >>>> Hello, >> >>>> >> >>>> Is there a way to trigger the on_kill() method of an operator through >> the >> >>>> UI? >> >>>> I tried changing the state of a running task and deleting it, but >> neither >> >>>> approach triggered the on_kill() method and so the process was still >> >>>> running. >> >>>> >> >>>> Similar question was asked on SO: >> >>>> https://stackoverflow.com/questions/43626844/kill-the- >> >>>> airflow-task-running-on-a-remote-location-through-airflow-ui >> >>>> >> >>>> Richard >> >>>> >> >>> >> > >> >> > > > -- > > Joy Gao > Software Engineer > 350 Convention Way, Suite 200 > Redwood City, CA 94063 > Mobile: 669-224-9305 > > Payments partner to the platform economy
