If you have a large number of dags, it can take some time before the scheduler processes them. This is determined by the following settings in the [scheduler] section:
# after how much time a new DAGs should be picked up from the filesystem min_file_process_interval = 0 dag_dir_list_interval = 300 Bolke > On 18 Aug 2017, at 21:26, Maxime Beauchemin <[email protected]> > wrote: > > In 1.8.1 I'm pretty sure that in the context of the scheduler, the DAGs are > fully re-evaluated in subprocesses which fixes the re-import > sys.module-related caching issue of earlier versions that created > staleness. 1.8.1 shouldn't have scheduler staleness issues. > > Max > > On Fri, Aug 18, 2017 at 6:09 AM, Boris Tyukin <[email protected]> wrote: > >> see how much time it takes to parse your dag, if you do any heavy >> operations like database calls, file pulls etc. I remember someone had an >> issue with web server because it took a while to parse one dag. By default >> airflow will check dag folder very frequently (unless you changed the >> timing in config). You can start both scheduler and web server manually and >> just watch the logs >> >> On Fri, Aug 18, 2017 at 9:03 AM, Sumit Maheshwari <[email protected]> >> wrote: >> >>> Thanks Boris, >>> >>> Seems like there is some issue in my Scheduler, which sometimes picks up >>> dag modification immediately and sometimes doesn't pickup at all. And >> when >>> Scheduler doesn't pickup new dag, UI also doesn't show it, i.e. no >> refresh >>> button would be available. I am trying to debug that issue with >> Scheduler, >>> but not sure if it's a real issue at all or something else. >>> >>> >>> >>> On Fri, Aug 18, 2017 at 6:16 PM, Boris Tyukin <[email protected]> >>> wrote: >>> >>>> we run 1.8.1 with no debug mode. New dags are picked up just fine >> without >>>> restarts. as well as modifications (adding new tasks or changing >>>> definitions of new dags). When I want to see dag updated on UI, I hit >>>> refresh button but this is just for UI to show it properly. No restarts >>>> needed in our experience with 1.8.1 >>>> >>>> On Fri, Aug 18, 2017 at 6:40 AM, Sumit Maheshwari < >>> [email protected]> >>>> wrote: >>>> >>>>> Hi All, >>>>> >>>>> We are trying to upgrade to Airflow ver 1.8.2 from 1.7.0 and found a >>>> couple >>>>> of issues, major ones are: >>>>> >>>>> 1. We used to run the webserver in debug mode (-d) so we don't need >> to >>>>> restart whenever we add/modify any dag. But with 1.8.2 debug mode >>> doesn't >>>>> have any effect and we need to restart it after any change in dags. >>>>> >>>>> 2. Scheduler used to pick up any new dag landed in dags folder, but >>> that >>>> is >>>>> not happening anymore and we required to restart the scheduler as >> well. >>>>> >>>>> Any help would be highly appreciated. >>>>> >>>>> Thanks, >>>>> Sumit >>>>> >>>> >>> >>
