[
https://issues.apache.org/jira/browse/AIRFLOW-1491?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16135194#comment-16135194
]
Nick Bauer commented on AIRFLOW-1491:
-------------------------------------
I don't think this is an airflow issue, but rather a celery config issue.
setting task_acks_late = True might solve this issue for you.
> Celery executor restarts result in duplicate tasks
> --------------------------------------------------
>
> Key: AIRFLOW-1491
> URL: https://issues.apache.org/jira/browse/AIRFLOW-1491
> Project: Apache Airflow
> Issue Type: Bug
> Reporter: George Leslie-Waksman
> Assignee: George Leslie-Waksman
>
> Since the Celery executor uses internal state to determine what it has queued
> already, anything that wipes the executor state (such as a scheduler restart)
> will result in rescheduling of tasks.
> This bug exacerbates a number of task scheduling race conditions and can
> result in duplicate task execution.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
