My quick guess - both running with local executor? That runs all af processes in one... so you’d have 2 schedulers.
It’s perfectly acceptable to run AF across a couple EC2 instances, but you have to select which parts of the stack run in which one, and support multiple airflow Configs in some cases. Sent from a device with less than stellar autocorrect > On Mar 28, 2019, at 3:11 PM, Adityan MS <[email protected]> wrote: > > Hi all, > > We have 2 EC2 instances running airflow 1.10.0 apps. As per my > understanding there should be only 1 airflow scheduler service running at a > given time. However lately we have started to see that mysteriously 2 > scheduler processes are running, which ends up causing a deadlock and our > Airflow tasks don't run. This ends up giving us an error in the > `scheduler.logs` - `Executor reports task instance <TaskInstance: <Task > Name> 2019-03-25 09:00:00+00:00 [queued]> finished (success) although the > task says its queued. Was the task killed externally?` > > To resolve this I go and manually kill the scheduler process by using `kill > -9 PID`. The scheduler failover controller takes care of spawning a new > scheduler process and all tasks resume running. Has anyone faced this > issue? How can I prevent 2 scheduler processes from coming up, and why do > they come up since we are not manually issuing a command to bring them up > in the first place? > > Thanks!
