[
https://issues.apache.org/jira/browse/AIRFLOW-4308?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16817266#comment-16817266
]
Jarek Potiuk commented on AIRFLOW-4308:
---------------------------------------
OK. I think I found the problem. In airflow code we are mixing
datetime.datetime usage and pendulum. This is not really good in case of python
3.6 because the datetime.fold() is set to 0 by default in python 3.6 and
pendulum backports fold calculation for python 3.5. So when we are crossing the
DST in 3.5 pendulum sets fold parameter to be = 1 (it takes the second instance
of the same time - looks like it intelligently guesses that it should be the
case).
In python 3.6 it takes the first instance because fold parameter is set to 0.
This can be demonstrated by running the following code:
{code:java}
from pendulum import timezone
from datetime import datetime
timezone = timezone('Europe/Zurich')
value = datetime(year=2018, month=10, day=28, hour=2, minute=00, second=0)
print(datetime.isoformat(value))
print(timezone.convert(value))
{code}
Result in python 3.5:
{code:java}
Python 3.5.4 (v3.5.4:3f56838976, Aug 7 2017, 12:56:33)
[GCC 4.2.1 (Apple Inc. build 5666) (dot 3)] on darwin
runfile('/Users/potiuk/Library/Preferences/IntelliJIdea2019.1/scratches/scratch_9.py',
wdir='/Users/potiuk/Library/Preferences/IntelliJIdea2019.1/scratches')
2018-10-28T02:00:00
2018-10-28 02:00:00+01:00{code}
Result in python 3.6:
{code:java}
Python 3.6.7 (v3.6.7:6ec5cf24b7, Oct 20 2018, 03:02:14)
[GCC 4.2.1 Compatible Apple LLVM 6.0 (clang-600.0.57)] on darwin
runfile('/Users/potiuk/Library/Preferences/IntelliJIdea2019.1/scratches/scratch_9.py',
wdir='/Users/potiuk/Library/Preferences/IntelliJIdea2019.1/scratches')
2018-10-28T02:00:00
2018-10-28 02:00:00+02:00
{code}
What solves it (and makes consistent) is to set fold parameter to 1 in python
3.6. Fold = 1 only matters when the clock is moved backwards, so it should not
be a problem for any other date/time:
{code:java}
from pendulum import timezone
from datetime import datetime
timezone = timezone('Europe/Zurich')
value = datetime(year=2018, month=10, day=28, hour=2, minute=00, second=0,
fold=1)
print(datetime.isoformat(value))
print(timezone.convert(value))
{code}
Result (Python 3.6 only as fold parameter is missing on 3.5):
{code:java}
Python 3.6.7 (v3.6.7:6ec5cf24b7, Oct 20 2018, 03:02:14)
[GCC 4.2.1 Compatible Apple LLVM 6.0 (clang-600.0.57)] on darwin
runfile('/Users/potiuk/Library/Preferences/IntelliJIdea2019.1/scratches/scratch_9.py',
wdir='/Users/potiuk/Library/Preferences/IntelliJIdea2019.1/scratches')
2018-10-28T02:00:00
2018-10-28 02:00:00+01:00
{code}
> Changed Time Zone behaviour in Python 3.6+ vs Python 3.5
> --------------------------------------------------------
>
> Key: AIRFLOW-4308
> URL: https://issues.apache.org/jira/browse/AIRFLOW-4308
> Project: Apache Airflow
> Issue Type: Bug
> Reporter: Jarek Potiuk
> Priority: Major
>
> It seems that one of the tests is failing in Python 3.5 vs. Python 3.6
> related to different behaviour of python with regards to TZ environment
> variable: [https://bugs.python.org/issue30062]
> We've yet have to determine whether this is a problem with the test or
> Airflow code:
>
> The test below works fine in Python 3.5.
> {code:java}
> + nosetests
> tests.models.test_dag:DagTest.test_following_previous_schedule_daily_dag_CEST_to_CET
> .
> ----------------------------------------------------------------------
> Ran 1 test in 0.005s
> OK
> {code}
>
> But it fails in Python 3.6:
> {code:java}
> nosetests
> tests.models.test_dag:DagTest.test_following_previous_schedule_daily_dag_CEST_to_CET
> F
> ======================================================================
> FAIL: Make sure DST transitions are properly observed
> ----------------------------------------------------------------------
> Traceback (most recent call last):
> File "/opt/airflow/tests/models/test_dag.py", line 800, in
> test_following_previous_schedule_daily_dag_CEST_to_CET
> self.assertEqual(next_local.isoformat(), "2018-10-28T03:00:00+01:00")
> AssertionError: '2018-10-28T02:00:00+01:00' != '2018-10-28T03:00:00+01:00'
> - 2018-10-28T02:00:00+01:00
> ? ^
> + 2018-10-28T03:00:00+01:00
> ? ^{code}
>
> Update: it is likely related to the differences how `fold` works differently
> in different python versions and not to the TZ bug:
> [https://pendulum.eustace.io/docs/#using-the-timezone-library-directly]
> And here is the python 3.6 change that triggers it most likely : Local Time
> disambiguation changed in Python 3.6:
> [https://docs.python.org/3/whatsnew/3.6.html#pep-495-local-time-disambiguation]
>
>
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
