Re: [PROPOSAL] Improved compatiblity checks for Providers (running unit tests for multiple airflow versions)

OK. Tests should be green now - all the issues are "handled" - there are few follow up tasks from the tests run on 2.9.1 but the PR should be quite ready for final review and merge now and I can attempt to look at 2.8 compatibility once it's done. On Mon, May 13, 2024 at 1:17 AM Ja

Re: [VOTE] Airflow Providers prepared on May 12, 2024

+1 (binding): verified reproducibility, signatures, checksums, licences. On Sun, May 12, 2024 at 9:34 PM Elad Kalif wrote: > Hey all, > > I have just cut the new wave Airflow Providers packages. This email is > calling a vote on the release, > which will last for 72 hours - which means that it

Re: [PROPOSAL] Improved compatiblity checks for Providers (running unit tests for multiple airflow versions)

that will need to be addressed before we release 2.1.10 so I need confirmation / comments from Niko/Daniel if my findings are correct. On Fri, May 10, 2024 at 12:54 PM Jarek Potiuk wrote: > > Just for clarification, this is only related to the provider's tests, > right? > > Absolutely. >

Re: [PROPOSAL] Improved compatiblity checks for Providers (running unit tests for multiple airflow versions)

t; right? > > > > On Fri, 10 May 2024 at 13:15, Jarek Potiuk wrote: > > > > Regarding Airflow 2.7 and Airflow 2.8 in the time we are ready to move > > forward to the initial version of Airflow 3 providers might already drop > > support of these versions in prov

Re: [PROPOSAL] Improved compatiblity checks for Providers (running unit tests for multiple airflow versions)

these versions in providers. > > Airflow 2.7 in the mid of August 2024 > > Airflow 2.8 in the mid of December 2024 > > > > > > > > On Fri, 10 May 2024 at 12:32, Jarek Potiuk wrote: > > > >> And yes - as we get down to 2.8 and 2.7 it might be possi

Re: [PROPOSAL] Improved compatiblity checks for Providers (running unit tests for multiple airflow versions)

and then 2.7 - so it might be that some of the refactors and changes will need to be applied to make it easier to maintain. On Fri, May 10, 2024 at 10:27 AM Jarek Potiuk wrote: > Yep. I think these are all good ideas, and I think this should be part of > our big Airflow 2 vs. Airflow 3 disc

Re: [PROPOSAL] Improved compatiblity checks for Providers (running unit tests for multiple airflow versions)

ers CI, > in another word it is no GA for the end users. This might be changed in the > future but let's focus that this package only for Airflow development > internals > > On Fri, 10 May 2024 at 01:08, Jarek Potiuk wrote: > > > Hello everyone, > > > > As part of p

[PROPOSAL] Improved compatiblity checks for Providers (running unit tests for multiple airflow versions)

Hello everyone, As part of preparation for the Airflow 3 move and (possible) provider separation (I have some ideas how to do it but that should be a separate discussion) I took on the task of improving our compatibility tests for Providers. I discussed it briefly with Kaxil and Ash and decided

Re: [VOTE] AIP-67 Multi-team deployment of Airflow components

Just to clarify the state for that one. I would like to put that one on-hold until we get clarity on Airflow 2 vs Airflow 3 approach: https://lists.apache.org/thread/3chvg9964zvh15mtrbl073f4oj3nlzp2 There is currently a veto from Ash, so until it is withdrawn or we change the problematic "team"

Re: [HUGE DISCUSSION] Airflow3 and tactical (Airflow 2) vs strategic (Airflow 3) approach

ts.apache.org/thread/7f6qyr8w2n8w34g63s7ybhzphgt8h43m> < > https://lists.apache.org/thread/7f6qyr8w2n8w34g63s7ybhzphgt8h43m> < > https://lists.apache.org/thread/7f6qyr8w2n8w34g63s7ybhzphgt8h43m;>. > This data > could be critical in ensuring our decisions are well

Re: [VOTE] Proposal for adding Telemetry via Scarf

leases). J. On Thu, May 9, 2024 at 6:46 AM Jarek Potiuk wrote: > +1 (binding) > > On Thu, May 9, 2024 at 4:47 AM Wei Lee wrote: > >> +1 non-binding >> >> Best, >> Wei >> >> > On May 9, 2024, at 10:39 AM, Phani Kumar >> > >> wrote:

Re: [VOTE] Proposal for adding Telemetry via Scarf

+1 (binding) On Thu, May 9, 2024 at 4:47 AM Wei Lee wrote: > +1 non-binding > > Best, > Wei > > > On May 9, 2024, at 10:39 AM, Phani Kumar > wrote: > > > > +1 binding, looking forward to add Scarf > > > > On Thu, May 9, 2024 at 7:42 AM Kaxil Naik wrote: > > > >> Hi all, > >> > >> Discussion

Re: [HUGE DISCUSSION] Airflow3 and tactical (Airflow 2) vs strategic (Airflow 3) approach

I am currently on sick leave, and still recovering - hoping to be able to travel next week to the US as planned, so I just wanted to break out of it to make one comment here. I got a clearer head now a bit with medications hopefully working. I am still taking it that should help me to get over

Re: [VOTE] Release Apache Airflow Python Client 2.9.0 from 2.9.0rc1

+1 (binding): tested reproducibility, signatures, checksums, licences (code is autogenerated), run some basic tests with Airflow 2.9.0 with the RC1 client On Tue, Apr 23, 2024 at 1:05 AM Jed Cunningham wrote: > Hey fellow Airflowers, > > I have cut the first release candidate for the Apache

Re: [HUGE DISCUSSION] Airflow3 and tactical (Airflow 2) vs strategic (Airflow 3) approach

quested feature in the Airflow User > Survey, > > 2.2 Modern UI - also comes up a lot > > 2.3 Different DAG distribution processes > > 2.4 Different execution mechanisms > > I know there are many more which I don't currently recall. > > > > 3. Airflow adoption >

Re: [DISCUSS] DRAFT AIP-68 Extended Plugin Interface + AIP-69 Remote Executor

Hey Jens, I looked at the AIPs when you created them and I very much like the directions put there - but it also got me into a lot of thinking on the future of Airflow and AIPs. See the thread I started https://lists.apache.org/thread/3chvg9964zvh15mtrbl073f4oj3nlzp2 - about Airflow 3. I think

Team id as dag_id prefix: (was [VOTE] AIP-67)

Hey Ash and Daniel (and others unconvinced), I split the discussion here so that we do not clutter the VOTE thread. I hope I can convince you that yes, it makes sense to approach it this way and that you withdraw the veto Ash. It needs a bit of context and the separate discussion I started on

[HUGE DISCUSSION] Airflow3 and tactical (Airflow 2) vs strategic (Airflow 3) approach

Hello here, I have been thinking a lot recently and discussing with some people and I am more and more convinced it's about the time we - as a community - should start doing changes considering "Airflow 2" current and "Airflow 3" future. *TL;DR: I think we should seriously start work on Airflow

Re: [DISCUSS] Consider disabling self-hosted runners for commiter PRs

from the maintainers if that helps with their experience. On Thu, Apr 18, 2024 at 10:59 AM Jarek Potiuk wrote: > PR switching it here: https://github.com/apache/airflow/pull/39106 - > sorry for the delay in following up on that one. > > J. > > On Fri, Apr 5, 2024 at 6:08 PM Wei Lee

[VOTE] AIP-67 Multi-team deployment of Airflow components

Hello here. I have not not heard a lot of feedback after my last update, so let me start a vote, hoping that the last changes proposed addressed most of the concerns. Just to recap. the proposal is here:

Re: [VOTE] Airflow Providers prepared on April 16, 2024

+1 (binding): checked reproducibility, licences, signatures, checksums - all look good. On Thu, Apr 18, 2024 at 12:27 PM Pankaj Singh wrote: > +1 (non-binding) tested my changes. > > Thanks > Pankaj > > > > On Wed, Apr 17, 2024 at 12:24 PM Amogh Desai > wrote: > > > +1 non binding > > > >

Re: [CALL FOR HELP] Help on Connexion 3 migration needed

efine what it is going to be? Dunno what others think :). But it would be good to hear what others think, as this one is one of the things that definitely holds us back. J. On Tue, Apr 16, 2024 at 10:36 PM Jarek Potiuk wrote: > I don't think AIP is needed (because it's not really a user facing c

Re: [DISCUSS] Consider disabling self-hosted runners for commiter PRs

. Plus, it saves some of the > cost. > > Best, > Wei > > > On Apr 5, 2024, at 11:36 PM, Jarek Potiuk wrote: > > > > Seeing no big "no's" - I will prepare and run the experiment - starting > > some time next week, after we get 2.9.0 out - I d

Re: [CALL FOR HELP] Help on Connexion 3 migration needed

ally if we will have feedback from those who know more about the involved technology stack. That's why I mostly opened this thread :) J. On Tue, Apr 16, 2024 at 5:15 PM Ryan Hatter wrote: > Does the scope of this PR warrant an AIP? > > On Tue, Apr 16, 2024 at 6:40 AM Jarek Potiuk

[CALL FOR HELP] Help on Connexion 3 migration needed

Hello here, I have a kind request for help from maintainers (and other contributors who are not maintainers) - on the Connexion 3 migration for Airflow. PR here (unfortunately - it's one big PR and cannot be split): https://github.com/apache/airflow/pull/39055. I would love some general comments

Re: [PROPOSAL] Keep >= LATEST MINOR for all providers using common.sql provider

ider, e.g. amazon, google, microsoft.azure > > > > > > > > If it is not a big deal I have no objections to adding this because I > > do > > > > not have a better solution which could be implemented "Here and Now". > > > > > > >

Re: [VOTE] Airflow Providers prepared on April 13, 2024

+1 (binding) for yandex: checked signatures, licences, checksums, reproducibility. On Sun, Apr 14, 2024 at 2:51 PM Elad Kalif wrote: > databricks provider is excluded from RC2 > let the vote continue only for yandex provider > > On Sun, Apr 14, 2024 at 8:10 AM Pankaj Koti > wrote: > > > -1

[DISCUSS] Redis licencing changes impact

Hello here, I've raised the discussion on private@ and it seems that there are no private/controversies there, so I bring the discussion to devlist where it belongs. You might want to be aware that Redis announced licensing changes that make future Redis 7.4+ releases not good for "mandatory"

Re: [VOTE] Airflow Providers prepared on April 10, 2024

+1 (binding). Checked all my changes, including the celery provider bug that prevented 3.6.1 from running on Airflow 2.7.* (celery 3.6.2rc1 now works just fine for Airflow 2.7). Checked reproducibility, licences, signatures, checksums. All good. On Wed, Apr 10, 2024 at 8:11 PM Vincent Beck

[PROPOSAL] Keep >= LATEST MINOR for all providers using common.sql provider

Hello here, I have a proposal to add a general policy that all our providers depending on common.sql provider always use >= LATEST_MNOR version of the provider. For example, following the change here https://github.com/apache/airflow/pull/38707 by David we will update all sql providers to have:

Re: [DISCUSS] Common.util provider?

how we can solve similar things in the future? Do we want it at all? It has some challenges - yes it DRY's the code but it also introduces coupling. J. On Sun, Mar 10, 2024 at 6:21 PM Jarek Potiuk wrote: > Coming back to it - what about the "polypill" :)? What's different v

Re: [DISCUSS] DRAFT AIP-67 Multi-tenant deployment of Airflow components

;GRPC API" - proposed by Daniel in one of the recent PRs, I think it's a better name and I started to use it everywhere. I believe that goes quite far in addressing all the concerns raised. I would love to start a vote on it by the end of the week unless there are serious concerns. Feel free

Re: [DISCUSS] Asynchronous SQLAlchemy

Yep. If we can make both Postgres and MySQL work with Async - I am also all for the "All" approach. If it means that we need to support only certain drivers and certain versions of the DBs - so be it. As mentioned in my original comments (long time ago when we had MSSQL support) - this was not

Re: [ANNOUNCE] New committer: Wei Lee

Congrats Wei! Indeed, well deserved! On Mon, Apr 8, 2024 at 10:54 AM Hussein Awala wrote: > Congrats Wei, well deserved! > > On Monday, April 8, 2024, Ash Berlin-Taylor wrote: > > > The Project Management Committee (PMC) for Apache Airflow has invited Wei > > Lee to become a committer and > >

Re: [VOTE] Release Airflow 2.9.0 from 2.9.0rc3

+1 (binding): checked reproducibility, licences, signatures, checksums, run a few dags, installed with celery executor and verified that configuration is properly displayed in UI (comparing with 2.9.0rc2) On Sun, Apr 7, 2024 at 8:16 AM Ephraim Anierobi wrote: > Hey fellow Airflowers, > > I have

Re: [DISCUSS] Consider disabling self-hosted runners for commiter PRs

38779 On Fri, Apr 5, 2024 at 4:21 PM Bishundeo, Rajeshwar wrote: > +1 with trying this out. I agree with keeping the canary builds > self-hosted in order to validate the usage for the PRs. > > -- Rajesh > > > From: Jarek Potiuk > Reply-To: "dev@airflow.apache.org"

Re: [DISCUSS] Consider disabling self-hosted runners for commiter PRs

ion. Do not > > > click links or open attachments unless you can confirm the sender and > > know > > > the content is safe. > > > > > > > > > > > > AVERTISSEMENT: Ce courrier électronique provient d’un expéditeur > externe. > >

Re: [VOTE] Release Airflow 2.9.0 from 2.9.0rc2

+1 (binding) - checked reproducibility, signatures, checksums, licences - > all good. Installed it, run a few dags, clicked through a number of screens. All looks good. Also verified the final package and it looks good with the right FAB >=1.0.2 dependency. All looks good. On Thu, Apr 4, 2024 at

[DISCUSS] Consider disabling self-hosted runners for commiter PRs

Hello everyone, TL;DR With some recent changes in GitHub Actions and the fact that ASF has a lot of runners available donated for all the builds, I think we could experiment with disabling "self-hosted" runners for committer builds. The self-hosted runners of ours have been extremely helpful

[ANNOUNCE] Apache Airflow Providers prepared on March 25, 2024 are released

Dear Airflow community, I'm happy to announce that new versions of Airflow Providers packages prepared on March 25. 2024 were just released. Full list of PyPI packages released is added at the end of the message. The source release, as well as the binary releases, are available here:

[RESULT][VOTE] Airflow Providers - release of March 25, 2024

Hello, [Filling-in for Elad - as we have a few follow-up tasks after we release FAB provider for Airflow 2.9.] Apache Airflow Providers (fab provider) prepared on March 25, 2024 have been accepted. 4 "+1" binding votes received: - Elad Kalif (binding) - Jarek Potiuk (binding) - Hus

Re: Apache Airflow 2.9.0b2 available for testing

, or to use different Python version and downgrade to Pendulum 2 On Fri, Mar 29, 2024 at 10:42 AM Jarek Potiuk wrote: > bound to Pendulum 3 of course. > > On Fri, Mar 29, 2024 at 10:42 AM Jarek Potiuk wrote: > >> Yes. IMHO we should add retroactively warning to 2.8.1 release not

Re: [DISCUSS] Proposal for adding Telemetry via Scarf

Hello everyone, it has to be: 1. Opt-in by default to not trigger security guys about new unplanned > activity after regular upgrade. > That's a very good point about security triggering Alexander, but I am not so sure it means that we "have to" do opt-in. There are other ways of communicating

Re: [DISCUSS] Proposal for adding Telemetry via Scarf

+1. All that sounds reasonable, there are precedents, ASF supports Scarf officially. Would be great to have access to such telemetry data. On Sat, Mar 30, 2024 at 1:18 AM Kaxil Naik wrote: > Hi all, > > I want to propose gathering telemetry for Airflow installations. As the > Airflow community,

Re: [VOTE] AIP-62 Getting Lineage from Hook Instrumentation

+1 (binding). I think that's a really important missing piece in lineage integration. On Thu, Mar 28, 2024 at 12:47 PM Maciej Obuchowski wrote: > Hello, > I would like to start a vote on AIP-62: Getting Lineage from Hook > Instrumentation, to be implemented into (presumably) Airflow 2.10. > >

Re: Apache Airflow 2.9.0b2 available for testing

bound to Pendulum 3 of course. On Fri, Mar 29, 2024 at 10:42 AM Jarek Potiuk wrote: > Yes. IMHO we should add retroactively warning to 2.8.1 release notes with > comment that you can still downgrade to Pendulum 2 to get old behaviour and > then in 2.9 release notes mention that 3

Re: Apache Airflow 2.9.0b2 available for testing

ink it's just nice to give a heads up to users. If you want to use > python 3.12 you are tied to pendulum 3 and airflow 2.9+. It's our first > release that supports 3.12 so I think it makes sense to add the note. > > B. > > Sent from my iPhone > > > On 28 Mar 2024,

Re: Apache Airflow 2.9.0b2 available for testing

> It is as we require pendulum 3 for python 3.12 support. Not really. Pendulum 3+ is the only version that works with Python 3.12, yes, But upgrading to Pendulum 3 was not connected to Airflow 2.9. Look it up in our repo. Pendulum <4 has been added in Airflow 2.8.1 and since then we have

Re: [DISCUSS] DRAFT AIP-67 Multi-tenant deployment of Airflow components

ted questions about > its > >> relation to "tenants." > >> > >> I also liked how you explained it in the AIP response. It's like when > >> Kubernetes talks about "multi-tenant" [2] and they mean it could be for > >> different cu

Re: [VOTE] March 2024 PR of the Month

+1 on Python 3.12 (#36755) - with the note that it's been a joint effort of quite a few people - Andrey, Bolke, Gopal Dirisao who contributed to it in Airflow, but also a number people who are maintainers of our dependencies: Israel Fruchter and Bret McGuire from Cassandra, Andreas Poehlmann

Re: [VOTE] AIP-64: Keep TaskInstance try history

+1 (binding) On Mon, Mar 25, 2024 at 10:51 PM Scheffler Jens (XC-AS/EAE-ADA-T) wrote: > +1 binding > > Sent from Outlook for iOS > > From: Pierre Jeambrun > Sent: Monday, March 25, 2024 9:19:36 PM > To: dev@airflow.apache.org > Subject:

Re: [VOTE] Airflow Providers prepared on March 25, 2024

+1 (binding): checked signatures, checksums, licences, reproducibility. I build latest main `airflow` (that will become b2) as `2.9.0rc1` package and installed it together with apache-airflow-providers-fab `1.0.2rc1` and they work nicely together. I was able to create users, authenticate, login

Re: [DISCUSS] Applying D105 rule for our codebase ("undocumented magic methods") ?

PR to remove it https://github.com/apache/airflow/pull/38452 On Mon, Mar 25, 2024 at 10:14 AM Jarek Potiuk wrote: > Ok. Seems we have a consensus :) > > On Mon, Mar 25, 2024 at 7:13 AM Wei Lee wrote: > >> +1 for removing this rule. If docstring is needed for some cases,

Re: [DISCUSS] Applying D105 rule for our codebase ("undocumented magic methods") ?

t; > >>>> > >>>> > >>>> > >>>> > >>>> On Wed, 20 Mar 2024 at 10:41, Pankaj Koti >> .invalid> > >>>> wrote: > >>>> > >>>>> +1 to what Aritra is saying. > >>>>>

Re: [VOTE] Release Apache Airflow Helm Chart 1.13.1 based on 1.13.1rc1

+1 binding. Verified reproducibility, signatures, licences, checksums. Installed it locally and got Airflow working. Though - one comment. While doing it now, the prometheus exporter image has not been pulled yet and the exporter is failing. The problem is that ` quay.io` is going through an

Re: Apache Airflow 2.9.0b1 available for testing

Just wanted to stress a note for this one: *apache-airflow-providers-fab==1.0.2b0 must be installed alongside* *this snapshot for Airflow to work.* (we will improve it in the next release) J. On Thu, Mar 21, 2024 at 1:12 PM Ephraim Anierobi wrote: > Hey fellow Airflowers, > > We have cut

Re: [VOTE] Release Airflow 2.8.4 from 2.8.4rc1

+1 (binding) - tested / verified all changes I was involved (either as fixer, bug introducer or both, particularly when both), verified reproducibility, licences, checksums, signatures, run a few DAGs - all looks good. On Wed, Mar 20, 2024 at 4:56 PM Jed Cunningham wrote: > Hey fellow

Re: Bad mixing of decorated and classic operators (users shooting themselves in their foot)

ry "heavy" for minimal benefit or > protection and limits what adventurous users can do) > > -a > > On 20 March 2024 07:05:25 GMT, Jarek Potiuk wrote: > >Just to add to the discussion - a discussion raised today > >https://github.com/apache/airflow/discussions/3831

Re: Python 3.12 support is here (!)

nstallable in "apache/airflow" image or "apache/airflow:2.9.0" - users will have to use "apache/airflow:2.9.0-python3.11" for example. J. On Mon, Mar 11, 2024 at 10:02 AM Jarek Potiuk wrote: > There were strange errors in the canary build which was not showi

Re: Bad mixing of decorated and classic operators (users shooting themselves in their foot)

is more of "I know there is an operator that does X, so > I will just use it inside the python function I invoke from the python > operator" - regardless of whether Jinja/templating becomes an issue or not. > > On Sat, Mar 9, 2024 at 9:06 PM Jarek Potiuk wrote: > > > I see

[DISCUSS] Applying D105 rule for our codebase ("undocumented magic methods") ?

Hey here, I wanted to quickly poll what people think about applying the https://docs.astral.sh/ruff/rules/undocumented-magic-method/ rule in our codebase. There are many uncontroversial rules - but that one is somewhat more controversial than others. See

Re: [VOTE] Remove experimental API

Would you still vote `-1` of course was the question. On Sat, Mar 16, 2024 at 8:37 PM Jarek Potiuk wrote: > Question: Jed, Ash: Would you still vote If we move it to provider (with > status "removed from maintenance except security fixes" - same as we did > with daskexecutor

Re: [VOTE] Remove experimental API

Question: Jed, Ash: Would you still vote If we move it to provider (with status "removed from maintenance except security fixes" - same as we did with daskexecutor? J. On Sat, Mar 16, 2024 at 8:25 PM Ash Berlin-Taylor wrote: > As much as I would love to remove it I'm with Jed: if it worked on

Re: [VOTE] Remove experimental API

+1 (binding) On Sat, Mar 16, 2024 at 1:14 PM Andrey Anshin wrote: > Greetings everyone! > > I would like to start a vote proces about removal of Experimental API > support into the next minor Airflow version, presumably 2.9, but it could > be postponed to 2.10. > > By default experimental REST

Re: [DISCUSS] DRAFT AIP-67 Multi-tenant deployment of Airflow components

etails, the AIP is available here - > https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-67+Multi-tenant+deployment+of+Airflow+components > > > Thanks > Shubham > Product Manager - Amazon MWAA > > > > *From: *Jarek Potiuk > *Reply-To: *"us...@airflow.apac

Re: [DISCUSS] DRAFT AIP-67 Multi-tenant deployment of Airflow components

=https%3A%2F%2Fcwiki.apache.org%2Fconfluence%2Fdisplay%2FAIRFLOW%2FAIP-67%2BMulti-tenant%2Bdeployment%2Bof%2BAirflow%2Bcomponents=05%7C02%7Cadam%40runbymany.com%7Ca5ad7fc69319431a5e7b08dc445bf26b%7C33632a8512c2443d9b064cfa3bf99965%7C0%7C0%7C638460408922503020%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAw

Re: [VOTE] AIP-59 Performance testing framework

+1 (binding) On Wed, Mar 13, 2024 at 7:40 AM Bartosz Jankiewicz wrote: > > Hi folks, > > The AIP for performance testing has been in review for quite some time and > I've included your feedback in the document. > > I'd like to call a vote, and if you agree I'd start a development of the >

[ANNOUNCEMENT] Updates to just released Airlfow 2.8.3 constraints and images

Hello everyone, As discussed earlier at the #release-management channel in Slack (and I am bringing it now o devlist) - the Airflow 2.8.3 images released yesterday had been updated today (following changes to constraints resulting from some early problem reports we found). In case you used a

Re: [DISCUSS] DRAFT AIP-67 Multi-tenant deployment of Airflow components

eployment+of+Airflow+components Feel free to comment here, or in the document. I would love to hear more voices, and have some ideas what to do next to validate the idea, so please - engage for now - but also expect some follow-ups. J. On Wed, Mar 6, 2024 at 9:16 AM Jarek Potiuk wrote: > Sooo.. Seem

Re: [PROPOSAL] Adding MSGraphSDK Async Operator to Airflow

triggerer and operator, with the > particularity it only works in async (e.g. deferred) mode. > > What do you think? > > Kind regards, > David > > -Original Message- > From: Jarek Potiuk > Sent: Sunday, 10 March 2024 18:19 > To: dev@airflow.apache.org > Cc: Bie

Re: Python 3.12 support is here (!)

There were strange errors in the canary build which was not showing up in the canary - so I need to revert and investigate. On Mon, Mar 11, 2024 at 8:52 AM Jarek Potiuk wrote: > And merged :) > > On Sun, Mar 10, 2024 at 6:52 PM Jarek Potiuk wrote: > >> Hey here, >> >

Re: Python 3.12 support is here (!)

And merged :) On Sun, Mar 10, 2024 at 6:52 PM Jarek Potiuk wrote: > Hey here, > > FINALLY - after 5 months (a little more than I initially anticipated - I > thought it will take 3-4 months) we can finally add Python 3.12. > > The time is about right - a day that we plan to cut

Python 3.12 support is here (!)

Hey here, FINALLY - after 5 months (a little more than I initially anticipated - I thought it will take 3-4 months) we can finally add Python 3.12. The time is about right - a day that we plan to cut the Airflow `v2-9-test` branch !! Thanks to Bolke for the final push on the Universal Pathlib

Re: [DISCUSS] Common.util provider?

Coming back to it - what about the "polypill" :)? What's different vs the "common.sql" way of doing it ? How do we think it can work ? On Thu, Feb 22, 2024 at 1:58 PM Jarek Potiuk wrote: > > The symbolic link approach seems to disregard all the external > prov

Re: [PROPOSAL] Adding MSGraphSDK Async Operator to Airflow

phSDKOperator could be based on the refactored HttpOperator, in > the meantime it would be nice if the operator would be already available in > Airflow > > What do you guys think? > > Kind regards, > David > > > -Original Message- > From: Jarek Potiuk > Sent

Re: [DISCUSS] DYNAMIC DAG RUNS WITH TASK PRIORITY

Also - not sure if you are subscribed to the devlist - so I will add your direct address here so that you can for sure see the answer (and if you are not subscribed, then by all means - do subscribe). On Sun, Mar 10, 2024 at 12:01 PM Jarek Potiuk wrote: > I think before writing

Re: [DISCUSS] DYNAMIC DAG RUNS WITH TASK PRIORITY

I think before writing AIP in confluence, I would encourage you to try to describe your idea in a shared google docs document and explain it. But before you do that - I'd encourage you to take a close look and deep dive into implementation of priorities. It might be different than you think, it

Re: Bad mixing of decorated and classic operators (users shooting themselves in their foot)

I see that we have already (thanks David!) a PR: https://github.com/apache/airflow/pull/37937 to forbid this use (which is cool and I am glad my discussion had some ripple effect :D ). I am quite happy to get this one merged once it passes tests/reviews, but I would still want to explore future

Re: [DISCUSS] Deprecation policy for apache-airflow-providers-google package

follow these rules to the same effect, but we prefer to be > transparent and inform the users of how the deprecation process will look. > The goal of the policy is to inform as opposed to enforce anything stricter > that we would like to do anyway. > > On Wed, Mar 6, 2024 at 5:1

Re: [VOTE] Release Airflow 2.8.3 from 2.8.3rc1

+1 (binding): checked reproducibility, sources, licences, checksums, run a few dags, tested all my changes, all looks good. One caveat. I think we still have the change that makes the default image `airflow/2.8.3rc1` point to `airflow/2.8.3rc1-python3.11` instead of `airfllow/2.8.3rc1-python3.8`.

[DISCUSS] DRAFT AIP-67 Multi-tenant deployment of Airflow components

Sooo.. Seems that it's an AIP time :D I've just published a Draft of AIP-67: Multi-tenant deployment of Airflow components https://cwiki.apache.org/confluence/display/AIRFLOW/%5BDRAFT%5D+AIP-67+Multi-tenant+deployment+of+Airflow+components This AIP is a bit lighter in detail than the others

Re: [VOTE] Airflow Providers prepared on March 04, 2024

+1 (binding): tested /checked my changes. Checked reproducibility, licences, signatures, checksums - all looks good. On Tue, Mar 5, 2024 at 5:51 PM Vincent Beck wrote: > > +1 non binding. All AWS system tests are running successfully against > apache-airflow-providers-amazon==8.19.0rc1. You

Re: [DISCUSS] Deprecation policy for apache-airflow-providers-google package

al policy for all providers, or that we should not codify our > > approach > > > at all since different providers have different requirements, instead of > > > having a policy per provider? > > > > > > I would be happy for this proposal (with some enhance

Re: Bad mixing of decorated and classic operators (users shooting themselves in their foot)

ks from the first page of the Google on "airflow slack notifications" > request: > > *Bad Examples:* > - > https://towardsdatascience.com/automated-alerts-for-airflow-with-slack-5c6ec766a823 > - > https://www.reply.com/data-reply/en/content/integrating-slack-alerts-in-airf

Re: [VOTE] Release Apache Airflow Helm Chart 1.13.0 based on 1.13.0rc1

+1 (binding): verified reproducibility (yay - works nicely finally !), signatures, checksums, licences. Installed Airflow 2.8.2 in kind cluster using the "rc1" - published chart., played a bit with it - ran a few DAGs using Breeze - using the same version as released in the Helm Chart - all looks

Re: [DISCUSS] Considering trying out uv for our CI workflows

image. End result are pretty much identical images (for size and looks like content - and they pass our PROD image tests - and airflow works as usual in them) J. On Tue, Feb 27, 2024 at 7:58 PM Jarek Potiuk wrote: > One more update - I am still looking at it and fine-tuning stuff and will >

Re: [DISCUSS] Deprecation policy for apache-airflow-providers-google package

lutely not. SemVer - is not a promise things won't break, it's a promise that our intention is that things won't break. But whether the way our users use it will make it break or not, that's another story. J On Thu, Feb 29, 2024 at 12:38 PM Jarek Potiuk wrote: > BTW. Another example of an

Re: [DISCUSS] Deprecation policy for apache-airflow-providers-google package

so - very much depending on nature of those deprecations) On Thu, Feb 29, 2024 at 12:36 PM Jarek Potiuk wrote: > I think it's a good idea to codify the rules, indeed what Elad mentioned, > we **mostly** follow a very similar approach, but it's not written > anywhere. > > >

Re: [DISCUSS] Deprecation policy for apache-airflow-providers-google package

I think it's a good idea to codify the rules, indeed what Elad mentioned, we **mostly** follow a very similar approach, but it's not written anywhere. Re: points of Elad: 1. I am ok with 6 months, but I agree here we should not have "strict" limits. It's fine to have some default (i.e.

CVE-2024-25128: Apache Airlfow Vulnerability: custom, long deprecated OpenID (NOT OIDC)

CVE-2024-25128: Vulnerability in custom, long deprecated OpenID (NOT OIDC) authentication method in Flask AppBuilder Severity: moderate Affected versions: - Apache Airflow before 2.8.2 Description: When Flask-AppBuilder configuration is set to ``AUTH_TYPE`` set to ``AUTH_OID``, it allows an

Updated pandas constraints for Airflow 2.8.2 (pandas==2.1.4)

Hey here, We've just solved a teething problem with Airflow 2.8.2 pandas dependency that aa-mathias (thanks Mattthias!) reported to us (Andrey fixed it in main already by adding < 2.2 requirement for pandas). The problem is that Pandas 2.2.1 has a problem with sqlalchemy 1.4 compatibility

Re: [DISCUSS] Considering trying out uv for our CI workflows

at if you've been using breeze and were sometimes afraid > to > > > hit "y" to rebuild the image, being afraid that it will take 20 minutes > or > > so - not any more. It should be WAY faster now. > > I'm very excited about this speed up as well as our CI :) > >

Re: [DISCUSS] Considering trying out uv for our CI workflows

Summarising where we are: After ~24 hrs of operations, it looks really cool and fulfills (and actually exceeds) all my expectations. * Multiple PRs succeeded, we got quite a few constraints updated automatically after successful canary runs:

Re: Bad mixing of decorated and classic operators (users shooting themselves in their foot)

stead. > > > > > > Regarding preventing this anti-pattern, I can think of two ways: > > > > > > Add a check in BaseOperator to detect whether __init__ is called in a > > task > > > work context, and error out guiding users to use hooks instead. > &

Re: [VOTE] January 2024 PR of the Month

Thanks TP, for bringing it :) If I can split my vote between #37101 and #36797 - > that would be it On Tue, Feb 27, 2024 at 6:19 AM Tzu-ping Chung wrote: > I want to nominate #36797.[1] > > This one has been a pretty popular request as well.[2] arguably dating > even further back since dynamic

Re: Bad mixing of decorated and classic operators (users shooting themselves in their foot)

instincts of our users guide us :) Would love to hear what others think J On Tue, Feb 27, 2024 at 8:09 AM Jarek Potiuk wrote: > Hello here, > > I have seen recently at least a few times that our users started to use a > strange pattern > > @task(task_id='some_id', provide_context

Bad mixing of decorated and classic operators (users shooting themselves in their foot)

Hello here, I have seen recently at least a few times that our users started to use a strange pattern @task(task_id='some_id', provide_context=True) def some_dummy_task(**context): ti = context['ti'] cmd2 = 'echo "pushing 2"' dbt_task = BashOperator(task_id='some_dummy_task_id',

Re: [VOTE] January 2024 PR of the Month

+1 on #37101 -> I think all the changes that make the dataset feature more powerful (and especially one that our users definitely asked for) is a really cool way to make our users look at Airflow through the lens of "modern and slick". This one is definitely taking us closer to this goal. On Mon,

Re: [DISCUSS] Considering trying out uv for our CI workflows

lp here. > > But one interesting idea that airflow could look at is periodically > running test cases against either "--resolution=lowest" or > "--resolution=lowest-direct" to try and flush out bad old dependencies? > > Damian > > -Original Message- &

Re: [ANNOYUNCEMENT] Fix to Airflow 2.8.1 / 2.8.2 constraints (smtp) - downgrade smtp provider to 1.5.0

Ok. turned out that image refresh was not needed because ... smtp provider is not included. Which is a bit surprising (will add it in the future I think) but also a bit easier as constraints got updated and point to 1.5.0 of smtp provider now. On Mon, Feb 26, 2024 at 4:02 PM Jarek Potiuk wrote

  1   2   3   4   5   6   7   8   9   10   >