I've done a bit more digging - the issue is of our tz-aware handling inside following_schedule (and previous schedule) - causing it to loop.
This section of the croniter docs seems relevant https://github.com/kiorky/croniter#about-dst Be sure to init your croniter instance with a TZ aware datetime for this to work !: >>> local_date = tz.localize(datetime(2017, 3, 26)) >>> val = croniter('0 0 * * *', local_date).get_next(datetime) I think the problem is that we are _not_ passing a TZ aware dag in and we should be. > On 30 Oct 2018, at 17:35, Bolke de Bruin <bdbr...@gmail.com> wrote: > > Oh that’s a great environment to start digging. Thanks. I’ll have a look. > > B. > > Verstuurd vanaf mijn iPad > >> Op 30 okt. 2018 om 18:25 heeft Ash Berlin-Taylor <a...@apache.org> het >> volgende geschreven: >> >> This line in airflow.jobs (line 874 in my checkout) is causing the loop: >> >> last_run = dag.get_last_dagrun(session=session) >> if last_run and next_run_date: >> while next_run_date <= last_run.execution_date: >> next_run_date = dag.following_schedule(next_run_date) >> >> >> >>> On 30 Oct 2018, at 17:20, Ash Berlin-Taylor <a...@apache.org> wrote: >>> >>> Hi, kaczors on gitter has produced a minmal reproduction case: >>> https://github.com/kaczors/airflow_1_10_tz_bug >>> >>> Rough repro steps: In a VM, with time syncing disabled, and configured with >>> system timezone of Europe/Zurich (or any other CEST one) run >>> >>> - `date 10280250.00` >>> - initdb, start scheduler, webserver, enable dag etc. >>> - `date 10280259.00` >>> - wait 5-10 mins for scheduler to catch up >>> - After the on-the-hour task run the scheduler will spin up another process >>> to parse the dag... and it never returns. >>> >>> I've only just managed to reproduce it, so haven't dug in to why yet. A >>> quick hacky debug print shows something is stuck in an infinite loop. >>> >>> -ash >>> >>>> On 29 Oct 2018, at 17:59, Bolke de Bruin <bdbr...@gmail.com> wrote: >>>> >>>> Can this be confirmed? Then I can have a look at it. Preferably with dag >>>> definition code. >>>> >>>> On the licensing requirements: >>>> >>>> 1. Indeed licensing header for markdown documents. It was suggested to use >>>> html comments. I’m not sure how that renders with others like PDF though. >>>> 2. The licensing notifications need to be tied to a specific version as >>>> licenses might change with versions. >>>> >>>> Cheers >>>> Bolke >>>> >>>> Verstuurd vanaf mijn iPad >>>> >>>>> Op 29 okt. 2018 om 12:39 heeft Ash Berlin-Taylor <a...@apache.org> het >>>>> volgende geschreven: >>>>> >>>>> I was going to make a start on the release, but two people have reported >>>>> that there might be an issue around non-UTC dags and the scheduler >>>>> changing over from Summer time. >>>>> >>>>>> 08:45 Emmanuel> Hi there, we are currently experiencing a very strange >>>>>> issue : we have hourly DAGs with a start_date in a local timezone (not >>>>>> UTC) and since (Sunday) the last winter time change they don’t run >>>>>> anymore. Any idea ? >>>>>> 09:41 <Emmanuel> it impacted all our DAG that had a run at 3am >>>>>> (Europe/Paris), the exact time of winter time change :( >>>>> >>>>> I am going to take a look at this today and see if I can get to the >>>>> bottom of it. >>>>> >>>>> Bolke: are there any outstanding tasks/issues that you know of that might >>>>> slow down the vote for a 1.10.1? (i.e. did we sort of out all the >>>>> licensing issues that were asked of us? I thought I read something about >>>>> license declarations in markdown files?) >>>>> >>>>> -ash >>>>> >>>>>> On 28 Oct 2018, at 14:46, Bolke de Bruin <bdbr...@gmail.com> wrote: >>>>>> >>>>>> I agree with that, but I would favor time based releases instead. We are >>>>>> again at the point that a release takes so much time that the gap is >>>>>> getting really big again. @ash why not start releasing now and move the >>>>>> remainder to 1.10.2? I dont think there are real blockers (although we >>>>>> might find them). >>>>>> >>>>>> >>>>>>> On 28 Oct 2018, at 15:35, airflowuser >>>>>>> <airflowu...@protonmail.com.INVALID> wrote: >>>>>>> >>>>>>> I was really hoping that >>>>>>> https://github.com/apache/incubator-airflow/pull/4069 will be merged >>>>>>> into 1.10.1 >>>>>>> Deleting dags was a highly requested feature for 1.10 - this can fix >>>>>>> the problem with it. >>>>>>> >>>>>>> >>>>>>> ‐‐‐‐‐‐‐ Original Message ‐‐‐‐‐‐‐ >>>>>>>>> On Friday, October 26, 2018 6:12 PM, Bolke de Bruin >>>>>>>>> <bdbr...@gmail.com> wrote: >>>>>>>> >>>>>>>> Hey Ash, >>>>>>>> >>>>>>>> I was wondering if you are picking up the 1.10.1 release? Master is >>>>>>>> speeding ahead and you were tracking fixes for 1.10.1 right? >>>>>>>> >>>>>>>> B. >>>>> >>> >>
