gs as soon and early as possible.
Damian
-Original Message-
From: Wei Lee
Sent: Tuesday, December 19, 2023 9:55 AM
To: dev@airflow.apache.org
Subject: Re: [DISCUSS] Future of Pendulum in Airflow
I’d vote for 2-a. Removing it in the upcoming minor release may not give enough
time for air
I’d vote for 2-a. Removing it in the upcoming minor release may not give enough
time for airflow users to adjust. On the other hand, we would also want to
decrease the amount of time we spend supporting various versions of Pendulum.
2.9 might be a reasonable compromise.
Best,
Wei
> On Dec 19,
I am for a) with deprecation warning when Pendulum 2 is installed in
Airflow 2.8.1+.
I understand we have a long way before we can get rid of Pendulum
dependency in general as it will be a major pain for our users for a
number of reasons, so I think getting rid of Pendulum is likely not
easy /
Greetings everyone!
Because pendulum 3 was released we probably need to discuss about support
for the previous version of pendulum in Airflow core.
There is not much options here
Option 1: Drop support of next patch version of Airflow, e.g. 2.8.1. It
might (or might not) break someone's
Pendulum 3 was released.
https://pypi.org/project/pendulum/
Bolke
On Fri, 1 Dec 2023 at 18:38, Aritra Basu wrote:
> I agree with wanting to stay away from having to deal with dst but I am on
> the side of biting the bullet and transitioning to datetime and testing
> thoroughly, especially
I agree with wanting to stay away from having to deal with dst but I am on
the side of biting the bullet and transitioning to datetime and testing
thoroughly, especially considering our current predicament with pendulum
and the potential of us being in the same place again potentially in the
Yeah. Agree it's a challenge but I think it's also an opportunity to
fix those - because whether we admit it, or not - we already have those
problems - regardless of Pendulum's use.
If you look at the issues (especially ones created around DST time) - there
are always one or two issues (even in
> So if we switch to the datetime + zone info + timezone we need to have a
test suite that verifies this.
We need to have these test suites anyway. According to the issues and
attempts to fix DST transitions we still have that problem even with use
Pendulum, or maybe it is happen because we use
The alternative is to vendor in or fork pendulum as part of the Airflow
project. Also a major undertaking but maybe less effort.
We shouldn't just address this as a technical challenge. The question becomes:
does that combination fornally do DST transitions correctly, does it do leap
seconds
Agree. I think this is the best way forward.- we have to bite the bullet of
potential backwards compatibility issues and separating out a compat layer
where we could optionally use pendulum instead would be the most "easy"
approach for our users. Most of them would not even notice if we do a good
IMHO, if we would like to replace pendulum to alternatives, the better
alternative would be combination of datetime.datetime + zoneinfo +
datetime.timezone
We still do hacks around pendulum functional, which makes hacks around
datetime.datetime, so in my perspective better remove redundant layers
"infamous fork of " should be "fork of infamous Akka" - just to be clear :)
On Wed, Nov 29, 2023 at 2:57 AM Jarek Potiuk wrote:
> It's a very rare occurrence that ASF accepts forks. Usually it has to be
> willingly donated by those who own the IP rights, - mainly because of
> trademark issues.
It's a very rare occurrence that ASF accepts forks. Usually it has to be
willingly donated by those who own the IP rights, - mainly because of
trademark issues. Licence is one thing, but carrying things like the name
of the project is not possible without clearing IP and donating the
project.
It
An option would be to fork Pendulum? It is MIT Licensed, I don't know
whether that poses problems to get in ASF?
If forking (?) which is somewhat non-ideal, would we want that 'in'
airflow?
If not 'in' airflow, I wonder if ASF incubator would accept a forked
project? [ anecdotally Linux
I have not heard back from maintainers, just a comment from someone else
who suggested donating a pendulum to the ASF (which is kinda interesting
idea). I followed up, let's see. I think if we do not hear back and there
will be another week or two where "Pendulum 3 supporting 3.12 support "is
Thanks Jarek
On Tue, Nov 21, 2023, 4:34 PM Jarek Potiuk wrote:
> I think we miss important insight - straight from the source.
>
> I believe it's time to be candid and simply ask questions for the future of
> Pendulum directly where we should - ie. we should just ask maintainers.
>
> I've just
I think we miss important insight - straight from the source.
I believe it's time to be candid and simply ask questions for the future of
Pendulum directly where we should - ie. we should just ask maintainers.
I've just started a very candid and open discussion there -
Just to clarify I'd like us to consider the possibility that no new
pendulum would be released or released at the end of 2024, like a
pessimistic scenario:
- What should we do in this case?
- Work out a backup plan.
Best Wishes
*Andrey Anshin*
On Fri, 17 Nov 2023 at 16:33, Jarek Potiuk
Also I think TP - had a document in the past (years ago) describing a
draft of a more complete alternative we can take to approach datetime vs.
pendulum dichotomy. I cannot easily find the document and discussion - but
I do remember it was proposing some interesting changes in the approach of
Hi,
I agree that the current speed of development of Pendulum leaves something
to be desired. However, I think we should not underestimate the effort of
replacing it. It is not just a matter of %s/pendulum/datetime/g so to say.
If we are *truly* thinking about moving to native datetime / zoneinfo
There is no changes in stable pendulum so let's try to continue this
discussion and start think about "Plan B"
Just a reminder:
- pendulum 2.1.2 released 3 years ago (at the time Airflow 1.10.x)
- pendulum 2 doesn't work well in Python 3.12, this is a showstopper for
the support Python 3.12
-
Good news, new pre-release of pendulum available:
https://pypi.org/project/pendulum/3.0.0b1/ ,
https://github.com/sdispater/pendulum/releases/tag/3.0.0b1
1. I checked and there is no memory leak on the unsuccessful parse.
2. Also check some timezone which previously doesn't work well, and as
; versions of Python.
>
> I appreciate this is a balancing issue, if pendulum doesn't make new
> releases then it can't be relied on forever.
>
> Damian
>
> -Original Message-----
> From: Bolke de Bruin
> Sent: Thursday, September 28, 2023 10:03 AM
> To: dev@airflow.apach
> We do not use pickling by default as we roll our own serialization
format. We probably just need the key (zoneinfo.key).
Yeah, for sure we don't need to rely on pickling, if I don't miss anything
than in Airflow 3 pickling serialisation should be removed. I just use as
reference info which
new releases
then it can't be relied on forever.
Damian
-Original Message-
From: Bolke de Bruin
Sent: Thursday, September 28, 2023 10:03 AM
To: dev@airflow.apache.org
Subject: Re: [DISCUSS] Future of Pendulum in Airflow
FYI:
I've just added:
https://github.com/apache/airflow/pull
FYI:
I've just added:
https://github.com/apache/airflow/pull/34667
which documents how to use newer timezone information with Pendulum.
Also work seems to be progressing (albeit slowly) on Pendulum 3:
https://github.com/sdispater/pendulum/issues/600#issuecomment-1711299677
Bolke
On Thu,
for serialization I am not too worried about ZoneInfo. We do not use
pickling by default as we roll our own serialization format. We probably
just need the key (zoneinfo.key).
I'm not sure what happened about this:
https://github.com/sdispater/pendulum/issues/590
Bolke
On Thu, 28 Sept 2023 at
I agree with all problems that you mention about datetime tz-aware data.
I lived for almost 30 years in a country which had in different periods of
time up to 10 time zones, and on a regular basis changed it (merge/unmerge)
, disable DST, temporarily enable DST. In addition I also worked in a
Thanks for starting the discussion Andrey.
Some background on the choice for Pendulum at the time. In the early days
of Airflow it wasn't timezone aware. Originating from Airbnb which had a
reasonable mature data organization the view was everything needs to be in
UTC. According to Maxime the
This discussion is more about the known problem of pendulum and how we
could deal with it and maybe how we (as Community) might help autor.
The library is mostly supported by a single author Sébastien Eustace (
https://github.com/sdispater) and it seems like we bump into the situation
which is
30 matches
Mail list logo
