From a user perspective my biggest concern is macros. There are many pendulum
methods that can, and are, being relied on.
And frankly time zone logic in the standard library is sub-optimal, any code
which involves manipulation in a particular time zone must also involve
converting to UTC and then back to that time zone again to avoid weird design
features (footguns).
Working in time zones with Airflow already requires very carefully date macro
construction, I have a module with several dozen Airflow date macros that
largely wrap around a base "today_macro":
> today_macro = "data_interval_end.in_tz(dag.timezone)"
> tomorrow_macro = f"macros.dates.next_day({today_macro})"
> ...
>
> today = f"{{{{{today_macro.format('YYYY-MM-DD')}}}}}"
> ...
These are imported from a common location in all our DAGs because date macro
writing had extremely high error rates. macros.dates points to very heavily
unit tested functions that have a lot of features, including arbitrary
calendars for business date calculations, and the logic relies on Pendulum
objects to correctly stay in, or convert to, the appropriate time zone.
That said, I agree the maintenance situation with Pendulum is dire and
untenable, but I just want to make it clear that for users which take advantage
of Airflow's time zone support it may be a non-trivial and error prone process
to convert to the standard library.
-----Original Message-----
From: Jarek Potiuk <ja...@potiuk.com>
Sent: Thursday, February 20, 2025 8:17 AM
To: dev@airflow.apache.org
Subject: Re: [DISCUSS] Python 3.13 support
One more finding which I miss..
* Pendulum - the elephant in the room (I like that phrase, yes).
I saw that Ash had already been involved in discussions about Python 3.13
support for pendulum in the PR that my friend from DLTHub started -
https://github.com/python-pendulum/pendulum/pull/871 - but regardless from that
we also need to have Python 3.13 wheels
https://github.com/python-pendulum/pendulum/issues/879#issuecomment-2671428096
. For now in my PR I had to install cargo (i.e. rust build system) in order to
compile it - and I had to install a very new one - because the one that comes
by default in Debian Bookworm, is too old to compile Pendulum.
And unfortunately - as we know from the past Pendulum is not really well
"maintained" - it took a long time for them to support 3.12 - and there does
not seem a lot of "responsibiity/responsiveness" of doing it from the
maintainers. On my personal (for now) list of "Airflow Beach Cleaning"
projects, Pendulum is quite firmly between "yellow" and "red" ("watch" vs.
"forego").
Being very blunt - should reconsider dropping Pendulum for Airflow 3 ? Or maybe
just make it optional and only used in Python < 3.13 - and introduce
potentially breaking changes by replacing Pendulum with non-naive datetime
everywhere ?
I think TP and Ash were the ones who knew most and discussed it before and
there was even a proposal from TP to remove the pendulum (as far as I
remember). Those changes would be potentially breaking, but because of
Pendulum' drop-in replacement for datetime - very little breaking.
WDYT? I will add it to the agenda of today's dev call -similarly as the other
"cleaning" topics I started recently.
J.
On Wed, Feb 19, 2025 at 7:26 PM Jarek Potiuk <ja...@potiuk.com> wrote:
> This is the time of the year when we attempt to see how far we are
> from supporting minor python version released in October (i.e. Python 3.13
> now).
> Usually it takes a couple of months before we can even try - because
> of the number of dependencies - and it was no different this time around.
>
> For now (it took me a better part of the day today) I managed to build
> the CI image with all dependencies and run `uv sync --all-extras
> --python 3.13 --reinstall` successfully (but with some cheating, so
> don't hold your breath yet). Also I bet there will be a number of test
> errors to fix, but for me the goal for now is to see how we stand with
> dependencies.
>
> For those who would like to see the changes - the PR is here
> https://github.com/apache/airflow/pull/46891
>
> And let me summarize the errors/issues/inconveniences I had to workaround:
>
> *The big one:*
>
> 1) Combination of Pandas, FAB and Sqlalchemy is such that we cannot
> have all of them installed in Python 3.12. We can have either Pandas +
> Sqlalchemy 2 (no Fab) or Fab and sqlalchemy 1 (no pandas).
>
> We know we need to upgrade to sqlalchemy 2 anyway - whatever the road
> we choose with Fab - either vendoring in and bumping Sqlalchemy 2 or
> hoping for Fab 5 release. For now - I simply removed Fab provider
> while the PR is in draft, to see if there are no more problems, and we
> will have to fix it before beta I am afraid anyway.
>
> 2) as explained in [1] I had to remove google-re and I used this
> opportunity to make an inventory of needed changes - I will summarize
> them in the google-re2 thread
>
> 3) Apache Beam is as usual a problem - this time there are two
> incompatibilities - pyarrow and pandas - but we can simply exclude
> beam from 3.13 (as usual) - usually they catch-up with the new version
> of python when the next is released (around october). They also have a
> lot of deps and more "holding" than ours.
>
> 4) databricks requires bumping of databricks-connector to 4.0.0 - we
> might need to hold on with that one depending on resulting of tests
> (this is also possible with bumping pandas + sqlalchemy and was not
> possible before, I do not expect major problems here)
>
> 5) Bumped grpcio-tools - they were held back previously - but removal
> of beam, upgrade of pandas allows it to be bumped to a newer version
> that supports 3.13 - that is helpful for a number of providers
> (waeviate,
> qdrant) to be 3.13 compatible..
>
> 6) Few small things removed: plyvel, ydb, possibly also yandex -
> depends on test results
>
> Currently there is a `uv` workspace bug that requires us to comment
> out rather than exclude python versions for dependencies, but I hope
> that - as usual - they will fix it in days rather than weeks. The
> issue in uv repo
> https://github.com/astral-sh/uv/issues/11629
>
> Those are all (non-trivial) issues so far, I think we are getting
> really, really close to having to decide on what to do with FAB and
> how to unleash sqlalchemy 2.
>
> I will talk to Daniel to get more informed about his 5.0.0 plans (he
> merged my security fix today in FAB and gearing for a new release
> apparently, but I bet it's going to be 4.5.4).
>
>
> Any comments?
>
> [1] https://lists.apache.org/thread/xtzdp4dx8s6dds4xdp9kdpohns9lvpst
>
________________________________
Strike Technologies, LLC (“Strike”) is part of the GTS family of companies.
Strike is a technology solutions provider, and is not a broker or dealer and
does not transact any securities related business directly whatsoever. This
communication is the property of Strike and its affiliates, and does not
constitute an offer to sell or the solicitation of an offer to buy any security
in any jurisdiction. It is intended only for the person to whom it is addressed
and may contain information that is privileged, confidential, or otherwise
protected from disclosure. Distribution or copying of this communication, or
the information contained herein, by anyone other than the intended recipient
is prohibited. If you have received this communication in error, please
immediately notify Strike at i...@striketechnologies.com, and delete and
destroy any copies hereof.
________________________________
CONFIDENTIALITY / PRIVILEGE NOTICE: This transmission and any attachments are
intended solely for the addressee. This transmission is covered by the
Electronic Communications Privacy Act, 18 U.S.C ''2510-2521. The information
contained in this transmission is confidential in nature and protected from
further use or disclosure under U.S. Pub. L. 106-102, 113 U.S. Stat. 1338
(1999), and may be subject to attorney-client or other legal privilege. Your
use or disclosure of this information for any purpose other than that intended
by its transmittal is strictly prohibited, and may subject you to fines and/or
penalties under federal and state law. If you are not the intended recipient of
this transmission, please DESTROY ALL COPIES RECEIVED and confirm destruction
to the sender via return transmittal.
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscr...@airflow.apache.org
For additional commands, e-mail: dev-h...@airflow.apache.org
