Not going into details - seems that Pendulum maintainers are much more
receptive than last time - (yay!). They already responded that they will
release the Python 3.13 version soon and I offered my help with testing it
with the Airflow CI test harness.
https://github.com/python-pendulum/pendulum/pull/871#issuecomment-2671761443
So at least in the case of Pendulum, I don't think we have time to change
anything for Airflow 3 - definitely the easiest way for us would be to
continue using Pendulum as we did.
On Thu, Feb 20, 2025 at 6:36 PM Damian Shaw <ds...@striketechnologies.com>
wrote:
> > The timezone conversion in stdlib is an intentional design choice. Not
> converting date times to matching timezones is a giant footgun that fires
> when you least expect it to. I did not like the design until I had the
> chance to listen to Paul Ganssle explain his decisions, and have to admit
> he made all the correct decisions.
>
> I appreciate it's an intentional choice, but this is a huge footgun:
> >>> datetime(2023, 3, 25, 22, tzinfo=ZoneInfo("Europe/Paris")) +
> timedelta(hours=8)
> datetime.datetime(2023, 3, 26, 6, 0,
> tzinfo=zoneinfo.ZoneInfo(key='Europe/Paris'))
>
> If one is not intermittently familiar with the stdlibs design choice then
> it's frankly near impossible to understand why it returns 6AM with Zoneinfo
> 'Europe/Paris', when it was in fact it was 7 AM on 2023-03-26 in Paris 8
> hours after 10 PM on 2023-03-25.
>
> Especially when current macros "just work":
> >>> pendulum.datetime(2023, 3, 25, 22, tz="Europe/Paris") +
> timedelta(hours=8)
> DateTime(2023, 3, 26, 7, 0, 0, tzinfo=Timezone('Europe/Paris'))
>
> I don't want to get in a discussion about design, I just want to point out
> from a user perspective of using Airflow this will be shocking, especially
> if macros might return a pendulum.datetime or a datetime.datetime under
> different contexts.
>
> -----Original Message-----
> From: Tzu-ping Chung <t...@astronomer.io.INVALID>
> Sent: Thursday, February 20, 2025 12:09 PM
> To: dev@airflow.apache.org
> Subject: Re: [DISCUSS] Python 3.13 support
>
> The timezone conversion in stdlib is an intentional design choice. Not
> converting date times to matching timezones is a giant footgun that fires
> when you least expect it to. I did not like the design until I had the
> chance to listen to Paul Ganssle explain his decisions, and have to admit
> he made all the correct decisions.
>
> With that said, I agree with you that Pendulum does provide a lot of
> convenient interfaces that the stdlib lacks. Everything has an equivalent,
> but stdlib is at times overly verbose. However, I don’t think that’s enough
> justification to depend on Pendulum. We can create our own datetime wrapper
> classes to do the same; it’s some maintenance overhead, but needing to
> chase an unstable target such as Pendulum is its own maintenance overhead.
> They cancel out.
>
> An alternative (to having our own wrapper class) is to make Pendulum an
> optional dependency. If you have it installed, we’ll wrap the datetime
> object in user-facing interfaces (e.g. template macros). That’s not too
> hard. We’ll simply not use it to implement Airflow itself.
>
> We’re probably too late to remove Pendulum usages in Airflow for 3.0, but
> I think 3.1 or 3.2 would be a reasonable target for either approach. If we
> do the wrapping part right, there should be no backward compatibility
> issues either.
>
> TP
>
>
> > On 21 Feb 2025, at 00:29, Damian Shaw <ds...@striketechnologies.com>
> wrote:
> >
> > From a user perspective my biggest concern is macros. There are many
> pendulum methods that can, and are, being relied on.
> >
> > And frankly time zone logic in the standard library is sub-optimal, any
> code which involves manipulation in a particular time zone must also
> involve converting to UTC and then back to that time zone again to avoid
> weird design features (footguns).
> >
> > Working in time zones with Airflow already requires very carefully date
> macro construction, I have a module with several dozen Airflow date macros
> that largely wrap around a base "today_macro":
> >> today_macro = "data_interval_end.in_tz(dag.timezone)"
> >> tomorrow_macro = f"macros.dates.next_day({today_macro})"
> >> ...
> >>
> >> today = f"{{{{{today_macro.format('YYYY-MM-DD')}}}}}"
> >> ...
> >
> > These are imported from a common location in all our DAGs because date
> macro writing had extremely high error rates. macros.dates points to very
> heavily unit tested functions that have a lot of features, including
> arbitrary calendars for business date calculations, and the logic relies on
> Pendulum objects to correctly stay in, or convert to, the appropriate time
> zone.
> >
> > That said, I agree the maintenance situation with Pendulum is dire and
> untenable, but I just want to make it clear that for users which take
> advantage of Airflow's time zone support it may be a non-trivial and error
> prone process to convert to the standard library.
> >
> > -----Original Message-----
> > From: Jarek Potiuk <ja...@potiuk.com>
> > Sent: Thursday, February 20, 2025 8:17 AM
> > To: dev@airflow.apache.org
> > Subject: Re: [DISCUSS] Python 3.13 support
> >
> > One more finding which I miss..
> >
> > * Pendulum - the elephant in the room (I like that phrase, yes).
> >
> > I saw that Ash had already been involved in discussions about Python
> > 3.13 support for pendulum in the PR that my friend from DLTHub started
> > -
> > https://github.com/python-pendulum/pendulum/pull/871 - but regardless
> > from that we also need to have Python 3.13 wheels
> > https://github.com/python-pendulum/pendulum/issues/879#issuecomment-26
> > 71428096 . For now in my PR I had to install cargo (i.e. rust build
> > system) in order to compile it - and I had to install a very new one -
> because the one that comes by default in Debian Bookworm, is too old to
> compile Pendulum.
> >
> > And unfortunately - as we know from the past Pendulum is not really well
> "maintained" - it took a long time for them to support 3.12 - and there
> does not seem a lot of "responsibiity/responsiveness" of doing it from the
> maintainers. On my personal (for now) list of "Airflow Beach Cleaning"
> > projects, Pendulum is quite firmly between "yellow" and "red" ("watch"
> vs.
> > "forego").
> >
> > Being very blunt - should reconsider dropping Pendulum for Airflow 3 ?
> Or maybe just make it optional and only used in Python < 3.13 - and
> introduce potentially breaking changes by replacing Pendulum with non-naive
> datetime everywhere ?
> >
> > I think TP and Ash were the ones who knew most and discussed it before
> and there was even a proposal from TP to remove the pendulum (as far as I
> remember). Those changes would be potentially breaking, but because of
> Pendulum' drop-in replacement for datetime - very little breaking.
> >
> > WDYT? I will add it to the agenda of today's dev call -similarly as the
> other "cleaning" topics I started recently.
> >
> >
> > J.
> >
> >
> >
> >
> >
> > On Wed, Feb 19, 2025 at 7:26 PM Jarek Potiuk <ja...@potiuk.com> wrote:
> >
> >> This is the time of the year when we attempt to see how far we are
> >> from supporting minor python version released in October (i.e. Python
> 3.13 now).
> >> Usually it takes a couple of months before we can even try - because
> >> of the number of dependencies - and it was no different this time
> around.
> >>
> >> For now (it took me a better part of the day today) I managed to
> >> build the CI image with all dependencies and run `uv sync
> >> --all-extras --python 3.13 --reinstall` successfully (but with some
> >> cheating, so don't hold your breath yet). Also I bet there will be a
> >> number of test errors to fix, but for me the goal for now is to see how
> we stand with dependencies.
> >>
> >> For those who would like to see the changes - the PR is here
> >> https://github.com/apache/airflow/pull/46891
> >>
> >> And let me summarize the errors/issues/inconveniences I had to
> workaround:
> >>
> >> *The big one:*
> >>
> >> 1) Combination of Pandas, FAB and Sqlalchemy is such that we cannot
> >> have all of them installed in Python 3.12. We can have either Pandas
> >> + Sqlalchemy 2 (no Fab) or Fab and sqlalchemy 1 (no pandas).
> >>
> >> We know we need to upgrade to sqlalchemy 2 anyway - whatever the
> >> road we choose with Fab - either vendoring in and bumping Sqlalchemy
> >> 2 or hoping for Fab 5 release. For now - I simply removed Fab
> >> provider while the PR is in draft, to see if there are no more
> >> problems, and we will have to fix it before beta I am afraid anyway.
> >>
> >> 2) as explained in [1] I had to remove google-re and I used this
> >> opportunity to make an inventory of needed changes - I will summarize
> >> them in the google-re2 thread
> >>
> >> 3) Apache Beam is as usual a problem - this time there are two
> >> incompatibilities - pyarrow and pandas - but we can simply exclude
> >> beam from 3.13 (as usual) - usually they catch-up with the new
> >> version of python when the next is released (around october). They
> >> also have a lot of deps and more "holding" than ours.
> >>
> >> 4) databricks requires bumping of databricks-connector to 4.0.0 - we
> >> might need to hold on with that one depending on resulting of tests
> >> (this is also possible with bumping pandas + sqlalchemy and was not
> >> possible before, I do not expect major problems here)
> >>
> >> 5) Bumped grpcio-tools - they were held back previously - but removal
> >> of beam, upgrade of pandas allows it to be bumped to a newer version
> >> that supports 3.13 - that is helpful for a number of providers
> >> (waeviate,
> >> qdrant) to be 3.13 compatible..
> >>
> >> 6) Few small things removed: plyvel, ydb, possibly also yandex -
> >> depends on test results
> >>
> >> Currently there is a `uv` workspace bug that requires us to comment
> >> out rather than exclude python versions for dependencies, but I hope
> >> that - as usual - they will fix it in days rather than weeks. The
> >> issue in uv repo
> >> https://github.com/astral-sh/uv/issues/11629
> >>
> >> Those are all (non-trivial) issues so far, I think we are getting
> >> really, really close to having to decide on what to do with FAB and
> >> how to unleash sqlalchemy 2.
> >>
> >> I will talk to Daniel to get more informed about his 5.0.0 plans (he
> >> merged my security fix today in FAB and gearing for a new release
> >> apparently, but I bet it's going to be 4.5.4).
> >>
> >>
> >> Any comments?
> >>
> >> [1] https://lists.apache.org/thread/xtzdp4dx8s6dds4xdp9kdpohns9lvpst
> >>
> > ________________________________
> > Strike Technologies, LLC (“Strike”) is part of the GTS family of
> companies. Strike is a technology solutions provider, and is not a broker
> or dealer and does not transact any securities related business directly
> whatsoever. This communication is the property of Strike and its
> affiliates, and does not constitute an offer to sell or the solicitation of
> an offer to buy any security in any jurisdiction. It is intended only for
> the person to whom it is addressed and may contain information that is
> privileged, confidential, or otherwise protected from disclosure.
> Distribution or copying of this communication, or the information contained
> herein, by anyone other than the intended recipient is prohibited. If you
> have received this communication in error, please immediately notify Strike
> at i...@striketechnologies.com, and delete and destroy any copies hereof.
> > ________________________________
> >
> > CONFIDENTIALITY / PRIVILEGE NOTICE: This transmission and any
> attachments are intended solely for the addressee. This transmission is
> covered by the Electronic Communications Privacy Act, 18 U.S.C ''2510-2521.
> The information contained in this transmission is confidential in nature
> and protected from further use or disclosure under U.S. Pub. L. 106-102,
> 113 U.S. Stat. 1338 (1999), and may be subject to attorney-client or other
> legal privilege. Your use or disclosure of this information for any purpose
> other than that intended by its transmittal is strictly prohibited, and may
> subject you to fines and/or penalties under federal and state law. If you
> are not the intended recipient of this transmission, please DESTROY ALL
> COPIES RECEIVED and confirm destruction to the sender via return
> transmittal.
> >
> > ---------------------------------------------------------------------
> > To unsubscribe, e-mail: dev-unsubscr...@airflow.apache.org
> > For additional commands, e-mail: dev-h...@airflow.apache.org
>
> ________________________________
> Strike Technologies, LLC (“Strike”) is part of the GTS family of
> companies. Strike is a technology solutions provider, and is not a broker
> or dealer and does not transact any securities related business directly
> whatsoever. This communication is the property of Strike and its
> affiliates, and does not constitute an offer to sell or the solicitation of
> an offer to buy any security in any jurisdiction. It is intended only for
> the person to whom it is addressed and may contain information that is
> privileged, confidential, or otherwise protected from disclosure.
> Distribution or copying of this communication, or the information contained
> herein, by anyone other than the intended recipient is prohibited. If you
> have received this communication in error, please immediately notify Strike
> at i...@striketechnologies.com, and delete and destroy any copies hereof.
> ________________________________
>
> CONFIDENTIALITY / PRIVILEGE NOTICE: This transmission and any attachments
> are intended solely for the addressee. This transmission is covered by the
> Electronic Communications Privacy Act, 18 U.S.C ''2510-2521. The
> information contained in this transmission is confidential in nature and
> protected from further use or disclosure under U.S. Pub. L. 106-102, 113
> U.S. Stat. 1338 (1999), and may be subject to attorney-client or other
> legal privilege. Your use or disclosure of this information for any purpose
> other than that intended by its transmittal is strictly prohibited, and may
> subject you to fines and/or penalties under federal and state law. If you
> are not the intended recipient of this transmission, please DESTROY ALL
> COPIES RECEIVED and confirm destruction to the sender via return
> transmittal.
>
