> The timezone conversion in stdlib is an intentional design choice. Not
> converting date times to matching timezones is a giant footgun that fires
> when you least expect it to. I did not like the design until I had the chance
> to listen to Paul Ganssle explain his decisions, and have to admit he made
> all the correct decisions.
I appreciate it's an intentional choice, but this is a huge footgun:
>>> datetime(2023, 3, 25, 22, tzinfo=ZoneInfo("Europe/Paris")) +
>>> timedelta(hours=8)
datetime.datetime(2023, 3, 26, 6, 0,
tzinfo=zoneinfo.ZoneInfo(key='Europe/Paris'))
If one is not intermittently familiar with the stdlibs design choice then it's
frankly near impossible to understand why it returns 6AM with Zoneinfo
'Europe/Paris', when it was in fact it was 7 AM on 2023-03-26 in Paris 8 hours
after 10 PM on 2023-03-25.
Especially when current macros "just work":
>>> pendulum.datetime(2023, 3, 25, 22, tz="Europe/Paris") + timedelta(hours=8)
DateTime(2023, 3, 26, 7, 0, 0, tzinfo=Timezone('Europe/Paris'))
I don't want to get in a discussion about design, I just want to point out from
a user perspective of using Airflow this will be shocking, especially if macros
might return a pendulum.datetime or a datetime.datetime under different
contexts.
-----Original Message-----
From: Tzu-ping Chung <t...@astronomer.io.INVALID>
Sent: Thursday, February 20, 2025 12:09 PM
To: dev@airflow.apache.org
Subject: Re: [DISCUSS] Python 3.13 support
The timezone conversion in stdlib is an intentional design choice. Not
converting date times to matching timezones is a giant footgun that fires when
you least expect it to. I did not like the design until I had the chance to
listen to Paul Ganssle explain his decisions, and have to admit he made all the
correct decisions.
With that said, I agree with you that Pendulum does provide a lot of convenient
interfaces that the stdlib lacks. Everything has an equivalent, but stdlib is
at times overly verbose. However, I don’t think that’s enough justification to
depend on Pendulum. We can create our own datetime wrapper classes to do the
same; it’s some maintenance overhead, but needing to chase an unstable target
such as Pendulum is its own maintenance overhead. They cancel out.
An alternative (to having our own wrapper class) is to make Pendulum an
optional dependency. If you have it installed, we’ll wrap the datetime object
in user-facing interfaces (e.g. template macros). That’s not too hard. We’ll
simply not use it to implement Airflow itself.
We’re probably too late to remove Pendulum usages in Airflow for 3.0, but I
think 3.1 or 3.2 would be a reasonable target for either approach. If we do the
wrapping part right, there should be no backward compatibility issues either.
TP
> On 21 Feb 2025, at 00:29, Damian Shaw <ds...@striketechnologies.com> wrote:
>
> From a user perspective my biggest concern is macros. There are many pendulum
> methods that can, and are, being relied on.
>
> And frankly time zone logic in the standard library is sub-optimal, any code
> which involves manipulation in a particular time zone must also involve
> converting to UTC and then back to that time zone again to avoid weird design
> features (footguns).
>
> Working in time zones with Airflow already requires very carefully date macro
> construction, I have a module with several dozen Airflow date macros that
> largely wrap around a base "today_macro":
>> today_macro = "data_interval_end.in_tz(dag.timezone)"
>> tomorrow_macro = f"macros.dates.next_day({today_macro})"
>> ...
>>
>> today = f"{{{{{today_macro.format('YYYY-MM-DD')}}}}}"
>> ...
>
> These are imported from a common location in all our DAGs because date macro
> writing had extremely high error rates. macros.dates points to very heavily
> unit tested functions that have a lot of features, including arbitrary
> calendars for business date calculations, and the logic relies on Pendulum
> objects to correctly stay in, or convert to, the appropriate time zone.
>
> That said, I agree the maintenance situation with Pendulum is dire and
> untenable, but I just want to make it clear that for users which take
> advantage of Airflow's time zone support it may be a non-trivial and error
> prone process to convert to the standard library.
>
> -----Original Message-----
> From: Jarek Potiuk <ja...@potiuk.com>
> Sent: Thursday, February 20, 2025 8:17 AM
> To: dev@airflow.apache.org
> Subject: Re: [DISCUSS] Python 3.13 support
>
> One more finding which I miss..
>
> * Pendulum - the elephant in the room (I like that phrase, yes).
>
> I saw that Ash had already been involved in discussions about Python
> 3.13 support for pendulum in the PR that my friend from DLTHub started
> -
> https://github.com/python-pendulum/pendulum/pull/871 - but regardless
> from that we also need to have Python 3.13 wheels
> https://github.com/python-pendulum/pendulum/issues/879#issuecomment-26
> 71428096 . For now in my PR I had to install cargo (i.e. rust build
> system) in order to compile it - and I had to install a very new one -
> because the one that comes by default in Debian Bookworm, is too old to
> compile Pendulum.
>
> And unfortunately - as we know from the past Pendulum is not really well
> "maintained" - it took a long time for them to support 3.12 - and there does
> not seem a lot of "responsibiity/responsiveness" of doing it from the
> maintainers. On my personal (for now) list of "Airflow Beach Cleaning"
> projects, Pendulum is quite firmly between "yellow" and "red" ("watch" vs.
> "forego").
>
> Being very blunt - should reconsider dropping Pendulum for Airflow 3 ? Or
> maybe just make it optional and only used in Python < 3.13 - and introduce
> potentially breaking changes by replacing Pendulum with non-naive datetime
> everywhere ?
>
> I think TP and Ash were the ones who knew most and discussed it before and
> there was even a proposal from TP to remove the pendulum (as far as I
> remember). Those changes would be potentially breaking, but because of
> Pendulum' drop-in replacement for datetime - very little breaking.
>
> WDYT? I will add it to the agenda of today's dev call -similarly as the other
> "cleaning" topics I started recently.
>
>
> J.
>
>
>
>
>
> On Wed, Feb 19, 2025 at 7:26 PM Jarek Potiuk <ja...@potiuk.com> wrote:
>
>> This is the time of the year when we attempt to see how far we are
>> from supporting minor python version released in October (i.e. Python 3.13
>> now).
>> Usually it takes a couple of months before we can even try - because
>> of the number of dependencies - and it was no different this time around.
>>
>> For now (it took me a better part of the day today) I managed to
>> build the CI image with all dependencies and run `uv sync
>> --all-extras --python 3.13 --reinstall` successfully (but with some
>> cheating, so don't hold your breath yet). Also I bet there will be a
>> number of test errors to fix, but for me the goal for now is to see how we
>> stand with dependencies.
>>
>> For those who would like to see the changes - the PR is here
>> https://github.com/apache/airflow/pull/46891
>>
>> And let me summarize the errors/issues/inconveniences I had to workaround:
>>
>> *The big one:*
>>
>> 1) Combination of Pandas, FAB and Sqlalchemy is such that we cannot
>> have all of them installed in Python 3.12. We can have either Pandas
>> + Sqlalchemy 2 (no Fab) or Fab and sqlalchemy 1 (no pandas).
>>
>> We know we need to upgrade to sqlalchemy 2 anyway - whatever the
>> road we choose with Fab - either vendoring in and bumping Sqlalchemy
>> 2 or hoping for Fab 5 release. For now - I simply removed Fab
>> provider while the PR is in draft, to see if there are no more
>> problems, and we will have to fix it before beta I am afraid anyway.
>>
>> 2) as explained in [1] I had to remove google-re and I used this
>> opportunity to make an inventory of needed changes - I will summarize
>> them in the google-re2 thread
>>
>> 3) Apache Beam is as usual a problem - this time there are two
>> incompatibilities - pyarrow and pandas - but we can simply exclude
>> beam from 3.13 (as usual) - usually they catch-up with the new
>> version of python when the next is released (around october). They
>> also have a lot of deps and more "holding" than ours.
>>
>> 4) databricks requires bumping of databricks-connector to 4.0.0 - we
>> might need to hold on with that one depending on resulting of tests
>> (this is also possible with bumping pandas + sqlalchemy and was not
>> possible before, I do not expect major problems here)
>>
>> 5) Bumped grpcio-tools - they were held back previously - but removal
>> of beam, upgrade of pandas allows it to be bumped to a newer version
>> that supports 3.13 - that is helpful for a number of providers
>> (waeviate,
>> qdrant) to be 3.13 compatible..
>>
>> 6) Few small things removed: plyvel, ydb, possibly also yandex -
>> depends on test results
>>
>> Currently there is a `uv` workspace bug that requires us to comment
>> out rather than exclude python versions for dependencies, but I hope
>> that - as usual - they will fix it in days rather than weeks. The
>> issue in uv repo
>> https://github.com/astral-sh/uv/issues/11629
>>
>> Those are all (non-trivial) issues so far, I think we are getting
>> really, really close to having to decide on what to do with FAB and
>> how to unleash sqlalchemy 2.
>>
>> I will talk to Daniel to get more informed about his 5.0.0 plans (he
>> merged my security fix today in FAB and gearing for a new release
>> apparently, but I bet it's going to be 4.5.4).
>>
>>
>> Any comments?
>>
>> [1] https://lists.apache.org/thread/xtzdp4dx8s6dds4xdp9kdpohns9lvpst
>>
> ________________________________
> Strike Technologies, LLC (“Strike”) is part of the GTS family of companies.
> Strike is a technology solutions provider, and is not a broker or dealer and
> does not transact any securities related business directly whatsoever. This
> communication is the property of Strike and its affiliates, and does not
> constitute an offer to sell or the solicitation of an offer to buy any
> security in any jurisdiction. It is intended only for the person to whom it
> is addressed and may contain information that is privileged, confidential, or
> otherwise protected from disclosure. Distribution or copying of this
> communication, or the information contained herein, by anyone other than the
> intended recipient is prohibited. If you have received this communication in
> error, please immediately notify Strike at i...@striketechnologies.com, and
> delete and destroy any copies hereof.
> ________________________________
>
> CONFIDENTIALITY / PRIVILEGE NOTICE: This transmission and any attachments are
> intended solely for the addressee. This transmission is covered by the
> Electronic Communications Privacy Act, 18 U.S.C ''2510-2521. The information
> contained in this transmission is confidential in nature and protected from
> further use or disclosure under U.S. Pub. L. 106-102, 113 U.S. Stat. 1338
> (1999), and may be subject to attorney-client or other legal privilege. Your
> use or disclosure of this information for any purpose other than that
> intended by its transmittal is strictly prohibited, and may subject you to
> fines and/or penalties under federal and state law. If you are not the
> intended recipient of this transmission, please DESTROY ALL COPIES RECEIVED
> and confirm destruction to the sender via return transmittal.
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: dev-unsubscr...@airflow.apache.org
> For additional commands, e-mail: dev-h...@airflow.apache.org
________________________________
Strike Technologies, LLC (“Strike”) is part of the GTS family of companies.
Strike is a technology solutions provider, and is not a broker or dealer and
does not transact any securities related business directly whatsoever. This
communication is the property of Strike and its affiliates, and does not
constitute an offer to sell or the solicitation of an offer to buy any security
in any jurisdiction. It is intended only for the person to whom it is addressed
and may contain information that is privileged, confidential, or otherwise
protected from disclosure. Distribution or copying of this communication, or
the information contained herein, by anyone other than the intended recipient
is prohibited. If you have received this communication in error, please
immediately notify Strike at i...@striketechnologies.com, and delete and
destroy any copies hereof.
________________________________
CONFIDENTIALITY / PRIVILEGE NOTICE: This transmission and any attachments are
intended solely for the addressee. This transmission is covered by the
Electronic Communications Privacy Act, 18 U.S.C ''2510-2521. The information
contained in this transmission is confidential in nature and protected from
further use or disclosure under U.S. Pub. L. 106-102, 113 U.S. Stat. 1338
(1999), and may be subject to attorney-client or other legal privilege. Your
use or disclosure of this information for any purpose other than that intended
by its transmittal is strictly prohibited, and may subject you to fines and/or
penalties under federal and state law. If you are not the intended recipient of
this transmission, please DESTROY ALL COPIES RECEIVED and confirm destruction
to the sender via return transmittal.
