[
https://issues.apache.org/jira/browse/AIRFLOW-5102?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16947007#comment-16947007
]
ASF subversion and git services commented on AIRFLOW-5102:
----------------------------------------------------------
Commit 68b8ec5f415795e4fa4ff7df35a3e75c712a7bad in airflow's branch
refs/heads/master from Ash Berlin-Taylor
[ https://gitbox.apache.org/repos/asf?p=airflow.git;h=68b8ec5 ]
[AIRFLOW-5102] Worker jobs should terminate themselves if they can't heartbeat
(#6284)
If a LocalTaskJob fails to heartbeat for
scheduler_zombie_task_threshold, it should shut itself down.
However, at some point, a change was made to catch exceptions inside the
heartbeat, so the LocalTaskJob thought it had managed to heartbeat
successfully.
This effectively means that zombie tasks don't shut themselves down.
When the scheduler reschedules the job, this means we could have two
instances of the task running concurrently.
> Workers fail to shutdown jobs after failed heartbeats
> -----------------------------------------------------
>
> Key: AIRFLOW-5102
> URL: https://issues.apache.org/jira/browse/AIRFLOW-5102
> Project: Apache Airflow
> Issue Type: Bug
> Components: worker
> Affects Versions: 1.10.3
> Reporter: Shantanu
> Assignee: Ash Berlin-Taylor
> Priority: Major
>
> If a LocalTaskJob fails to heartbeat for scheduler_zombie_task_threshold, it
> should shut itself down:
> [https://github.com/apache/airflow/blob/f34e13a/airflow/jobs/local_task_job.py#L109]
>
> However, at some point, a change was made to catch exceptions inside the
> heartbeat:
> [https://github.com/apache/airflow/blob/f34e13a/airflow/jobs/base_job.py#L194]
> LocalTaskJob now thinks heartbeats always succeed.
>
> This effectively means that zombie tasks don't shut themselves down. When the
> scheduler reschedules the job, this means we could have two instances of the
> task running concurrently.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
