When you see the task being launched again on startup of the executor, does
the second launched task get killed when it tries to startup or does it
actually run the `execute` task instance method (and try to launch a pod on
kubernetes).
I ask because the latter should not happen if postgres is
Hmm, perhaps we've just had a couple of bad/unlucky runs but, in general,
the underlying task-kill process doesn't really seem to work, from what
we've seen. I would guess this is related to
https://issues.apache.org/jira/browse/AIRFLOW-1623.
On Sun, Dec 17, 2017 at 12:22 PM, Bolke de Bruin
Shorter heartbeats, you might still have some tasks being scheduled
nevertheless due to the time window. However, if the tasks detects it is
running somewhere else, it should also terminate itself.
[scheduler]
# Task instances listen for external kill signal (when you clear tasks
# from the CLI
> P.S. I am assuming that you are talking about your scheduler going down,
not workers
Correct (and, in some unfortunate scenarios, everything else...)
> Normally a task will detect (on the heartbeat interval) whether its state
was changed externally and will terminate itself.
Hmm, that would
Quite important to know is, is that Airflow’s executors do not keep state after
a restart. This particularly affects distributed executors (celery, dask) as
the workers are independent from the scheduler. Thus at restart we reset all
the tasks in the queued state that the executor does not know
Upon further internal discussion, we might be seeing the task cloning
because the postgres DB is getting into a corrupted state...but unclear.
If consensus is we *shouldn't* be seeing this behavior, even as-is, we'll
push more on that angle.
On Sun, Dec 17, 2017 at 10:45 AM, Christopher Bockman
Hi all,
We run DAGs, and sometimes Airflow crashes (for whatever reason--maybe
something as simple as the underlying infrastructure going down).
Currently, we run everything on Kubernetes (including Airflow), so the
Airflow pods crashes generally will be detected, and then they will restart.
