1.10.1 isn't out yet, so 1.10.0 :) I think this only affects days with a schedule interval that is one hour or more frequently, and where the dag timezone is not UTC.
For anyone stuck on this in prod I think the fix is to manually create (via backfill or trigger) a dag run after the tz change over time. This should unblock the scheduler. Ash On 30 October 2018 18:43:30 GMT, David Klosowski <dav...@thinknear.com> wrote: >Hi Airflow Devs: > >Is this timezone issue in Airflow version 1.10.0 or only in 1.10.1? > >Thanks. > >Regards, >David > >On Tue, Oct 30, 2018 at 11:11 AM Bolke de Bruin <bdbr...@gmail.com> >wrote: > >> we specifically remove timezone info to determine the next schedule. >Ie. >> cron sets exact date times so tz info should not make sense. I’m >going to >> have a look now. >> >> >> > On 30 Oct 2018, at 19:01, Ash Berlin-Taylor <a...@apache.org> wrote: >> > >> > I've done a bit more digging - the issue is of our tz-aware >handling >> inside following_schedule (and previous schedule) - causing it to >loop. >> > >> > This section of the croniter docs seems relevant >> https://github.com/kiorky/croniter#about-dst >> > >> > Be sure to init your croniter instance with a TZ aware datetime >for >> this to work !: >> >>>> local_date = tz.localize(datetime(2017, 3, 26)) >> >>>> val = croniter('0 0 * * *', local_date).get_next(datetime) >> > >> > I think the problem is that we are _not_ passing a TZ aware dag in >and >> we should be. >> > >> >> On 30 Oct 2018, at 17:35, Bolke de Bruin <bdbr...@gmail.com> >wrote: >> >> >> >> Oh that’s a great environment to start digging. Thanks. I’ll have >a >> look. >> >> >> >> B. >> >> >> >> Verstuurd vanaf mijn iPad >> >> >> >>> Op 30 okt. 2018 om 18:25 heeft Ash Berlin-Taylor <a...@apache.org> >het >> volgende geschreven: >> >>> >> >>> This line in airflow.jobs (line 874 in my checkout) is causing >the >> loop: >> >>> >> >>> last_run = dag.get_last_dagrun(session=session) >> >>> if last_run and next_run_date: >> >>> while next_run_date <= last_run.execution_date: >> >>> next_run_date = >dag.following_schedule(next_run_date) >> >>> >> >>> >> >>> >> >>>> On 30 Oct 2018, at 17:20, Ash Berlin-Taylor <a...@apache.org> >wrote: >> >>>> >> >>>> Hi, kaczors on gitter has produced a minmal reproduction case: >> https://github.com/kaczors/airflow_1_10_tz_bug >> >>>> >> >>>> Rough repro steps: In a VM, with time syncing disabled, and >> configured with system timezone of Europe/Zurich (or any other CEST >one) >> run >> >>>> >> >>>> - `date 10280250.00` >> >>>> - initdb, start scheduler, webserver, enable dag etc. >> >>>> - `date 10280259.00` >> >>>> - wait 5-10 mins for scheduler to catch up >> >>>> - After the on-the-hour task run the scheduler will spin up >another >> process to parse the dag... and it never returns. >> >>>> >> >>>> I've only just managed to reproduce it, so haven't dug in to why >yet. >> A quick hacky debug print shows something is stuck in an infinite >loop. >> >>>> >> >>>> -ash >> >>>> >> >>>>> On 29 Oct 2018, at 17:59, Bolke de Bruin <bdbr...@gmail.com> >wrote: >> >>>>> >> >>>>> Can this be confirmed? Then I can have a look at it. Preferably >with >> dag definition code. >> >>>>> >> >>>>> On the licensing requirements: >> >>>>> >> >>>>> 1. Indeed licensing header for markdown documents. It was >suggested >> to use html comments. I’m not sure how that renders with others like >PDF >> though. >> >>>>> 2. The licensing notifications need to be tied to a specific >version >> as licenses might change with versions. >> >>>>> >> >>>>> Cheers >> >>>>> Bolke >> >>>>> >> >>>>> Verstuurd vanaf mijn iPad >> >>>>> >> >>>>>> Op 29 okt. 2018 om 12:39 heeft Ash Berlin-Taylor ><a...@apache.org> >> het volgende geschreven: >> >>>>>> >> >>>>>> I was going to make a start on the release, but two people >have >> reported that there might be an issue around non-UTC dags and the >scheduler >> changing over from Summer time. >> >>>>>> >> >>>>>>> 08:45 Emmanuel> Hi there, we are currently experiencing a >very >> strange issue : we have hourly DAGs with a start_date in a local >timezone >> (not UTC) and since (Sunday) the last winter time change they don’t >run >> anymore. Any idea ? >> >>>>>>> 09:41 <Emmanuel> it impacted all our DAG that had a run at >3am >> (Europe/Paris), the exact time of winter time change :( >> >>>>>> >> >>>>>> I am going to take a look at this today and see if I can get >to the >> bottom of it. >> >>>>>> >> >>>>>> Bolke: are there any outstanding tasks/issues that you know of >that >> might slow down the vote for a 1.10.1? (i.e. did we sort of out all >the >> licensing issues that were asked of us? I thought I read something >about >> license declarations in markdown files?) >> >>>>>> >> >>>>>> -ash >> >>>>>> >> >>>>>>> On 28 Oct 2018, at 14:46, Bolke de Bruin <bdbr...@gmail.com> >> wrote: >> >>>>>>> >> >>>>>>> I agree with that, but I would favor time based releases >instead. >> We are again at the point that a release takes so much time that the >gap is >> getting really big again. @ash why not start releasing now and move >the >> remainder to 1.10.2? I dont think there are real blockers (although >we >> might find them). >> >>>>>>> >> >>>>>>> >> >>>>>>>> On 28 Oct 2018, at 15:35, airflowuser ><airflowu...@protonmail.com.INVALID> >> wrote: >> >>>>>>>> >> >>>>>>>> I was really hoping that >> https://github.com/apache/incubator-airflow/pull/4069 will be merged >into >> 1.10.1 >> >>>>>>>> Deleting dags was a highly requested feature for 1.10 - this >can >> fix the problem with it. >> >>>>>>>> >> >>>>>>>> >> >>>>>>>> ‐‐‐‐‐‐‐ Original Message ‐‐‐‐‐‐‐ >> >>>>>>>>>> On Friday, October 26, 2018 6:12 PM, Bolke de Bruin < >> bdbr...@gmail.com> wrote: >> >>>>>>>>> >> >>>>>>>>> Hey Ash, >> >>>>>>>>> >> >>>>>>>>> I was wondering if you are picking up the 1.10.1 release? >Master >> is speeding ahead and you were tracking fixes for 1.10.1 right? >> >>>>>>>>> >> >>>>>>>>> B. >> >>>>>> >> >>>> >> >>> >> > >> >>
