ASF subversion and git services commented on AIRFLOW-224:

Commit fb89276750d23fc19a2d57036bc59d3aef846a26 in incubator-airflow's branch 
refs/heads/master from [~bolke]
[ https://git-wip-us.apache.org/repos/asf?p=incubator-airflow.git;h=fb89276 ]

[AIRFLOW-224] Collect orphaned tasks and reschedule them

Tasks can get orphaned if the scheduler is killed in the middle
of processing the tasks or if the MQ queue is cleared without
a worker having picked these up. Now tasks do not get set
to a scheduled state anymore if they have not been sent to the
executor yet. Next to that a garbage collector scans the executor
for tasks not being present and reschedules those if needed.

> Collect orphaned tasks in case of unclean shutdown
> --------------------------------------------------
>                 Key: AIRFLOW-224
>                 URL: https://issues.apache.org/jira/browse/AIRFLOW-224
>             Project: Apache Airflow
>          Issue Type: Improvement
>          Components: scheduler
>            Reporter: Bolke de Bruin
>             Fix For: Airflow 1.8
> Tasks in a "scheduled" state can get orphaned if either the executor or the 
> scheduler is shutdown uncleanly (kill -9). 
> This should be addressed by making sure the scheduler confirms sending tasks 
> to the executor and checking weather "scheduled" tasks are still in the queue 
> of the executor.

This message was sent by Atlassian JIRA

Reply via email to