see how much time it takes to parse your dag, if you do any heavy operations like database calls, file pulls etc. I remember someone had an issue with web server because it took a while to parse one dag. By default airflow will check dag folder very frequently (unless you changed the timing in config). You can start both scheduler and web server manually and just watch the logs
On Fri, Aug 18, 2017 at 9:03 AM, Sumit Maheshwari <[email protected]> wrote: > Thanks Boris, > > Seems like there is some issue in my Scheduler, which sometimes picks up > dag modification immediately and sometimes doesn't pickup at all. And when > Scheduler doesn't pickup new dag, UI also doesn't show it, i.e. no refresh > button would be available. I am trying to debug that issue with Scheduler, > but not sure if it's a real issue at all or something else. > > > > On Fri, Aug 18, 2017 at 6:16 PM, Boris Tyukin <[email protected]> > wrote: > > > we run 1.8.1 with no debug mode. New dags are picked up just fine without > > restarts. as well as modifications (adding new tasks or changing > > definitions of new dags). When I want to see dag updated on UI, I hit > > refresh button but this is just for UI to show it properly. No restarts > > needed in our experience with 1.8.1 > > > > On Fri, Aug 18, 2017 at 6:40 AM, Sumit Maheshwari < > [email protected]> > > wrote: > > > > > Hi All, > > > > > > We are trying to upgrade to Airflow ver 1.8.2 from 1.7.0 and found a > > couple > > > of issues, major ones are: > > > > > > 1. We used to run the webserver in debug mode (-d) so we don't need to > > > restart whenever we add/modify any dag. But with 1.8.2 debug mode > doesn't > > > have any effect and we need to restart it after any change in dags. > > > > > > 2. Scheduler used to pick up any new dag landed in dags folder, but > that > > is > > > not happening anymore and we required to restart the scheduler as well. > > > > > > Any help would be highly appreciated. > > > > > > Thanks, > > > Sumit > > > > > >
