Thanks for the detailed debugging! It gives us a point where to look at. B.
Verstuurd vanaf mijn iPad > Op 29 mei 2019 om 08:49 heeft Emmanuel Brard > <[email protected]> het volgende geschreven: > > Hi, > > We have noticed the same thing. On way for us to unblock it was to restart > the scheduler (there is a function that check missing tasks in executor at > the scheduler start-up phase). > We implemented a health-check that checks for these kind of stuck queued > tasks, but obviously fixing it in airflow core would be better. > > Cheers, > E > > On Wed, May 29, 2019 at 8:01 AM [email protected] <[email protected]> > wrote: > >> Hi All, >> We are observing intermittently that Tasks get stuck in queued state and >> never get executed by Airflow. >> On debugging it we found that one of the queued dependency was not met due >> to which task did not move from queued to running state. So task remained >> in queued state. >> (are_dependencies_met function returned false for QUEUE_DEPS inside >> _check_and_change_state_before_execution) >> >> By looking into scheduler code it seems that scheduler does not reschedule >> the queued state tasks due to which task never got added to executor queue >> again and remained stuck in queued state. >> >> There is a logic inside _check_and_change_state_before_execution function >> to move the task from queued to None state(which gets picked by scheduler >> for rescheduling) if RUN_DEPS are not met but this logic seems to be >> missing for QUEUE_DEPS. >> >> It seems that task should be moved to None state even if QUEUE_DEPS are >> not met. >> Any help/pointer on this would be useful. >> >> Thanks, >> Raman Gupta >> >> >> > > -- > > > > > > > > > GetYourGuide AG > > Stampfenbachstrasse 48 > > 8006 Zürich > > Switzerland > > > > <https://www.facebook.com/GetYourGuide> > <https://twitter.com/GetYourGuide> > <https://www.instagram.com/getyourguide/> > <https://www.linkedin.com/company/getyourguide-ag> > <http://www.getyourguide.com> > > > > > > >
