In 1.8.1 I'm pretty sure that in the context of the scheduler, the DAGs are fully re-evaluated in subprocesses which fixes the re-import sys.module-related caching issue of earlier versions that created staleness. 1.8.1 shouldn't have scheduler staleness issues.
Max On Fri, Aug 18, 2017 at 6:09 AM, Boris Tyukin <[email protected]> wrote: > see how much time it takes to parse your dag, if you do any heavy > operations like database calls, file pulls etc. I remember someone had an > issue with web server because it took a while to parse one dag. By default > airflow will check dag folder very frequently (unless you changed the > timing in config). You can start both scheduler and web server manually and > just watch the logs > > On Fri, Aug 18, 2017 at 9:03 AM, Sumit Maheshwari <[email protected]> > wrote: > > > Thanks Boris, > > > > Seems like there is some issue in my Scheduler, which sometimes picks up > > dag modification immediately and sometimes doesn't pickup at all. And > when > > Scheduler doesn't pickup new dag, UI also doesn't show it, i.e. no > refresh > > button would be available. I am trying to debug that issue with > Scheduler, > > but not sure if it's a real issue at all or something else. > > > > > > > > On Fri, Aug 18, 2017 at 6:16 PM, Boris Tyukin <[email protected]> > > wrote: > > > > > we run 1.8.1 with no debug mode. New dags are picked up just fine > without > > > restarts. as well as modifications (adding new tasks or changing > > > definitions of new dags). When I want to see dag updated on UI, I hit > > > refresh button but this is just for UI to show it properly. No restarts > > > needed in our experience with 1.8.1 > > > > > > On Fri, Aug 18, 2017 at 6:40 AM, Sumit Maheshwari < > > [email protected]> > > > wrote: > > > > > > > Hi All, > > > > > > > > We are trying to upgrade to Airflow ver 1.8.2 from 1.7.0 and found a > > > couple > > > > of issues, major ones are: > > > > > > > > 1. We used to run the webserver in debug mode (-d) so we don't need > to > > > > restart whenever we add/modify any dag. But with 1.8.2 debug mode > > doesn't > > > > have any effect and we need to restart it after any change in dags. > > > > > > > > 2. Scheduler used to pick up any new dag landed in dags folder, but > > that > > > is > > > > not happening anymore and we required to restart the scheduler as > well. > > > > > > > > Any help would be highly appreciated. > > > > > > > > Thanks, > > > > Sumit > > > > > > > > > >
