Hi, We have noticed the same thing. On way for us to unblock it was to restart the scheduler (there is a function that check missing tasks in executor at the scheduler start-up phase). We implemented a health-check that checks for these kind of stuck queued tasks, but obviously fixing it in airflow core would be better.
Cheers, E On Wed, May 29, 2019 at 8:01 AM [email protected] <[email protected]> wrote: > Hi All, > We are observing intermittently that Tasks get stuck in queued state and > never get executed by Airflow. > On debugging it we found that one of the queued dependency was not met due > to which task did not move from queued to running state. So task remained > in queued state. > (are_dependencies_met function returned false for QUEUE_DEPS inside > _check_and_change_state_before_execution) > > By looking into scheduler code it seems that scheduler does not reschedule > the queued state tasks due to which task never got added to executor queue > again and remained stuck in queued state. > > There is a logic inside _check_and_change_state_before_execution function > to move the task from queued to None state(which gets picked by scheduler > for rescheduling) if RUN_DEPS are not met but this logic seems to be > missing for QUEUE_DEPS. > > It seems that task should be moved to None state even if QUEUE_DEPS are > not met. > Any help/pointer on this would be useful. > > Thanks, > Raman Gupta > > > -- GetYourGuide AG Stampfenbachstrasse 48 8006 Zürich Switzerland <https://www.facebook.com/GetYourGuide> <https://twitter.com/GetYourGuide> <https://www.instagram.com/getyourguide/> <https://www.linkedin.com/company/getyourguide-ag> <http://www.getyourguide.com>
