Re: [DISCUSS] Proposal around the Injection of Task Execution Secrets

about a different name for this AIP) > On 7 Jun 2024, at 19:25, Ash Berlin-Taylor wrote: > >> IMHO - if we do not want to support DB access at all from workers, > triggerrers and DAG file processors, we should replace the current "DB" > bound interface with a n

Re: [DISCUSS] Number of queries to Airflow database in "DAG File Processing Stats"

> On 14 Jun 2024, at 10:22, Jarek Potiuk wrote: > > think in the future of Airflow 3 where > we will have task isolation, having `0` for all the DAGs will be a > prerequisite for switching to "task isolation" mode and this could be > actually verified in a migration tool. I think we still

Re: [DISCUSS] Number of queries to Airflow database in "DAG File Processing Stats"

We have `DAG Code` model in the database — which from memory (I haven’t checked just now) is the entire source of the file, even if there are multiple dags defined in one file, so we could add the columns to that row. -ash > On 14 Jun 2024, at 11:37, Jarek Potiuk wrote: > > Yep . Per DAG

Re: [DISCUSS] Proposal around the Injection of Task Execution Secrets

;> interface to >>and from Airflow Task User code to the Airflow system components >> including >>the meta-database, Airflow Executor, etc. >>2. >> >>Disable all direct database interaction from the Airflow Workers >> including

Re: [DISCUSS] AIP-71 Generalizing DAG Loader and Processor for Ephemeral Storage

> Non DAG, Non module assets as part of the DAG folder are out of scope. So say for example for some reason you include a GIF. This will not automatically be available without changes to your code. What about SQL files a task uses, either as a template or via something else such as dbt? How

Re: [VOTE] AIP-69 Remote Executor

Can we have a link to the pr please? The AIP doc itself is still light on what changes are actually needed On 18 May 2024 14:56:57 BST, Aritra Basu wrote: >+1 (non-binding) >The proposal was a good read, would love to see it come up and would love >to help out if you need a helping hand. > >--

Re: [HUGE DISCUSSION] Airflow3 and tactical (Airflow 2) vs strategic (Airflow 3) approach

> That would require some mechanism of declaring prior to task execution what > connections would be used That’s exactly what I’m proposing in the proposal doc I’m working on (It’s part of also overhauling and re-designing the “Task Execution interface” that also gives us the ability to nicely

Re: [HUGE DISCUSSION] Airflow3 and tactical (Airflow 2) vs strategic (Airflow 3) approach

> > > 1. *Core Architectural Changes:* We’re looking at foundational changes >> > > with Airflow 3—like redefining task priorities, separating task >> > > definition >> > > and task execution, and new AIPs like DAG versioning. remote execution >> >

Re: [HUGE DISCUSSION] Airflow3 and tactical (Airflow 2) vs strategic (Airflow 3) approach

There's a lot of technical debt hiding in Airflow, especially the scheduler that makes it harder and harder to efficiently add new features. At some point, very soon, we are going to have to remove some very infrequently used back compat shims that negatively affect performance. Without doing

Re: Refactor Scheduler Timed Events to be Async?

One thing to bear in mind here is the number of db connections - each connection can only be used by one thread or coroutine at a time, so even when the scheduler is changed to use async db calls, we might not be able to do a lot of the scheduled tasks concurrently. At least not without

Re: [VOTE] AIP-67 Multi-team deployment of Airflow components

It doesn’t affect my vote on the API, but I am very strongly against this one part of the AIP: > … dag_id are namespaced with `:` prefix. This specific part is getting an implementation/code veto from me. We made the mistake of overloading one column to store multiple things in Airflow before,

Re: [DISCUSS] Asynchronous SQLAlchemy

I’m all in favour of async SQLAlchemy. We’ve built two products exclusively at @ Astronomer that use sqlalchemy+psycopg3+async and love it. Async does take a bit of a learning curve, but SQLA has done it nicely and it works really well. I think this needs to be an all or nothing thing — having

[ANNOUNCE] New committer: Wei Lee

The Project Management Committee (PMC) for Apache Airflow has invited Wei Lee to become a committer and we are pleased to announce that they have accepted. Wei has been contributing for a number of months he also participated a lot in discussions and decisions on many aspects of Airflow but

Re: [DISCUSS] DRAFT AIP-67 Multi-tenant deployment of Airflow components

Thanks very much Jarek and folks for working on this proposal. I think we can ultimately get Airflow better as a result, but there was something that wasn’t sitting quite right with me, and I think I’ve finally managed to articulate it in to words. (I left this as a comment on the wiki AIP

Re: Bad mixing of decorated and classic operators (users shooting themselves in their foot)

The reason users are sure they can use operators like that is that it has worked for a long time - hell I even wrote a custom nested operator in the past (pre 2.0 admittedly). So this pr should only be a warning by default, or a config option to warn but not error Alternatively do we just

Re: [DISCUSS] Applying D105 rule for our codebase ("undocumented magic methods") ?

I'm for not enforcing this rule - as others have said its very unlikely to result in more useful docs for developers or end users. -asg On 20 March 2024 08:12:40 GMT, Andrey Anshin wrote: >±0 from my side > >Maybe we have to review all current methods which do not follow this rule >to find a

Re: [VOTE] Remove experimental API

As much as I would love to remove it I'm with Jed: if it worked on 2.0 it should work on all 2.x My vote is -1 On 16 March 2024 19:08:13 GMT, Jed Cunningham wrote: >I forgot to add the "why" - I view this as a breaking change still.

Re: [VOTE] "Require conversation resolution" setting in PRs as permanent solution

-1 On 6 February 2024 06:38:04 GMT, Bolke de Bruin wrote: >-1, binding. > > >Sent from my iPhone > >> On 5 Feb 2024, at 21:07, Scheffler Jens (XC-AS/EAE-ADA-T) >> wrote: >> >> +1 binding >> >> Sent from Outlook for iOS >> >> From:

Re: Idea for Discussion: custom TI dependencies

This is an absolute minefield of performance issues as this is the very centre of the hot path for the scheduler, so I would advise against any casual use of this. That said: I think it’s possible today. Set `deps` on your operator class like

Re: [VOTE] AIP 61 - Hybrid Executors

+1 but may I make a small(?) suggestion: If we also swap the alias and class name order around: From: 'LocalExecutor,my.custom.module.Executor' To: 'default=LocalExecutor, shortname=my.custom.module.Executor' This way opens up the option to have two different instances of the same executor

Re: [ANNOUNCE] Starting experimenting with "Require conversation resolution" setting

ot;why this way?" or "what about case Y?") If GH let discussions be resolved without also collapsing them I'd be +1, but mixing the two mens I prefer _not_ resolving discussions. -a On 31 January 2024 11:00:11 GMT, Ash Berlin-Taylor wrote: >I'm a -1 on keeping this as I don't

Re: [ANNOUNCE] Starting experimenting with "Require conversation resolution" setting

I'm a -1 on keeping this as I don't see it gives us any real benefit other than a rubber-stamp. Let's treat people as intelligent grown ups instead of children who need strict rules. On 31 January 2024 09:37:50 GMT, Pankaj Koti wrote: >+1 to keep this > >@Bolke de Bruin: I am just thinking

Re: [DISCUSS] "Require conversation resolution" in our PRs before merge?

Given the slow down over the holidays I don't think two weeks will be enough - make it 4? On 19 December 2023 20:33:23 GMT, Jarek Potiuk wrote: >Ash: > >> I.e. Convention over enforcement and treating people as mature adults not >children who need guard rails. > >I think it's quite the

Re: [DISCUSS] "Require conversation resolution" in our PRs before merge?

Weak -1 from me, because I don't think this needs to be enforced/required part of the workflow. I.e. Convention over enforcement and treating people as mature adults not children who need guard rails. -ash On 19 December 2023 13:12:05 GMT, Jarek Potiuk wrote: >Hey everyone, > >TL;DR; I have

Re: [DISCUSS] Airflow UI DAG Composer

AIP > <https://cwiki.apache.org/confluence/display/AIRFLOW/Airflow+Improvement+Proposals>, > just need to request edit access to it. And my Wiki ID is: *yuleili*. Thanks > > Best, > Yulei > > On Fri, Nov 10, 2023 at 4:16 AM Ash Berlin-Taylor wrote: > >> P

Re: [PROPOSAL] Deprecate URI Connection representation in favor of JSON

-1 — for the simple case the URI format is perfect: AIRFLOW_CONN_MY_DB=postgresql://user:pass@host We can push JSON format for more complex cases, and/or limit URI to only the simple cases as long as we don’t remove it entirely. -ash > On 17 Nov 2023, at 22:44, Andrey Anshin wrote: > >

Re: [DISCUSS] Airflow UI DAG Composer

Please make the document readable by anyone with the link. Thanks, Ash > On 9 Nov 2023, at 21:10, Yulei Li wrote: > > Hi Airflow community, > > My name is Yulei and I'm a software engineer from the Workflow Platform > team at Pinterest. On behalf of the Workflow Platform team and the >

Re: [ANNOUNCE] New committer: Jens Scheffler

✨✨徭 On 8 November 2023 08:41:42 GMT, Pierre Jeambrun wrote: >Congrats Jens! > >Le mer. 8 nov. 2023 à 09:40, Pankaj Koti >a écrit : > >> Many congratulations, Jens!!   >> >> >> Best regards, >> >> *Pankaj Koti* >> Senior Software Engineer (Airflow OSS Engineering team) >> Location: Pune,

Re: [VOTE] Restore dag_run.conf UI triggering option for 2.7.0

+100 The vote doesn't say the release must happen, just that it can happen. The final decision on when to release is still up to the release manager. On 12 August 2023 18:36:13 BST, Jarek Potiuk wrote: >Hello everyone, > >I would like to raise a vote about modifying the result of vote from

Re: [VOTE] The daskexecutor, cncf.provider, celery providers in 2.7.0. as regular not-preinstalled providers

Option a. _ash On 29 July 2023 19:25:28 BST, Jarek Potiuk wrote: >Hello everyone, > >I would like to resume the vote discussed last week whether we should >pre-install or not the three providers bringing the new executors. > >Most of the discussion happened in

Re: Allowing providers to load the respective default connections

I'm for changing it. How about `db migrate` as the only (non-deprecated) command we keep? On 27 July 2023 08:01:32 BST, Jarek Potiuk wrote: >Also one comment here: the whole point here is to remove confusion to >the users. If we leave `upgrade` in place, the confusion will remain. >So I think

Re: Allowing providers to load the respective default connections

How about `airflow connections create-default` instead of putting it under `db` — that way the DB subcommand is now exclusively about migrations, but connection management is all under the `connections` sub-command -ash > On 26 Jul 2023, at 07:51, Akash Sharma <2akash111...@gmail.com> wrote:

Re: [VOTE] Make (soon coming) dask provider preinstalled

-1 - based on the premise that your had to install the `dask` extra in the first place to get dask module of the right version, so if we make the existing extra depends on the new provider then it's good enough. On 21 July 2023 06:22:28 BST, Jarek Potiuk wrote: >Q: Do we want to pre-install

Re: [VOTE] Airflow Providers prepared on May 24, 2023

+1 binding On 24 May 2023 18:07:30 BST, Phani Kumar wrote: >+1 non binding . Tested microsoft.azure: 6.1.1rc1 > >and >it works as expected > >On Wed, May 24, 2023 at 9:16 PM Pankaj Koti > wrote: > >> +1 (non-binding)

Re: [VOTE] (extended) on AIP-50 (part 2) to finalize it

What they said - option B for me as well On 23 May 2023 15:49:58 BST, Collin McNulty wrote: >I also support Option B. +1 B, non-binding. > >Collin McNulty > >On Tue, May 23, 2023 at 9:33 AM Constance Martineau > wrote: > >> Hello, >> >> I think Option B is reasonable. +1 for B, non-binding. >>

Re: [VOTE] Airflow Providers prepared on May 16, 2023

-1 (binding) to at least anything with a logging provider: https://github.com/apache/airflow/pull/30994 is causing problems with custom logging config -- it leads to a circular import error Traceback (most recent call last): File

Re: AIP-56 Extensible user management

> - I am fine with (and actually I like it better) removing User and Role > related Rest APIs and CLI commands from Airflow. That will indeed make the > AIP way simpler and shorter I don't think we can do that: we still need to consider users when they are just getting started out with Airflow

Re: [VOTE] Release Airflow 2.6.0 from 2.6.0rc5

+1 binding. I'm having flashbacks to the 1.9.0 release. -ash On Apr 29 2023, at 2:00 pm, Jarek Potiuk wrote: > +1 binding. Checked licences, signatures, checksum, tested a few dags and > generally looked through the UI. All looks good. > > On Sat, Apr 29, 2023 at 1:02 PM Ephraim Anierobi >

Re: [VOTE] Release Airflow 2.6.0 from 2.6.0rc4

+1 (binding). Fourth time's the charm. :crossed-fingers: On Apr 28 2023, at 3:27 pm, Elad Kalif wrote: > +1 (binding) > ran dags locally works great > > On Fri, Apr 28, 2023 at 5:13 PM Jarek Potiuk wrote: > > +1 (binding) - checked signatures, checksums, licences. Installed and > > tested it

Re: [DISCUSS] Exclude some providers that hold us back from releasing

Yeah agreed. Each provider should be treated like a separate project/release - it's just that we batch up releases to save time of the release manager. -a On 28 March 2023 07:05:13 BST, Daniel Standish wrote: >It seems reasonable to me. > >On Mon, Mar 27, 2023 at 12:02 AM Jarek Potiuk

Re: [DISCUSS] AIP-52 updates - setup / teardown tasks

mething very bad happens and in that >case you want to stop all processing and leave the current state alone so >you can debug it. Is there any good reason to forbid this? I think no. >As stewards of this pipeline writing language, we should have sensible >defaults and maintain a good a

Re: [DISCUSS] AIP-52 updates - setup / teardown tasks

If the set-up ran then the short circuit shouldn't be able to skip it: take for example creating a cluster - you still want to delete it at the end even if you skipped all the other tasks! This is precisely what I mean by set up and tear down tasks being special! On 27 March 2023 04:02:32 BST,

Re: [DISCUSS] AIP-52 updates - setup / teardown tasks

>> absolutely essential that our choices do not cause trouble for multiple >> setup / teardowns in the future. So even if we postpone enablement of >> multiple, or put it behind an experimental flag, I do not think we can >> postpone *consideration* of multiple -- before we release this I think we

Re: [DISCUSS] a cache for Airflow Variables

I second Elad's view here. I would also propose an alternative fix: let's come up with a way to tell Airflow to not continuously reparse a file! A strawman example: ``` from airflow import ReparseMode, DAG AIRFLOW_REPARSE = ReparseMode.ON_FILE_CHANGED with DAG(...): ``` We could

Re: [DISCUSS] AIP-52 updates - setup / teardown tasks

I'm obviously in favour of the way the AIP was written, and that's for two primary reasons. 1. It's analogous to setup and teardown in testing frameworks where you don't ever explicitly call them - the framework handles it for you. 2. `task1 >> task2 >> teardown_task` to me falsely implies that

Re: [DISCUSS] Airflow - New SLA AIP

Hi Sung, Thanks for collecting all this one place, it's a great summary of the problem with SLAs. After 2.6 was out one of the next things we (with my "one of the Astronomer people who work on Airflow" hat on here) wanted to look at SLAs in general, and specifically extending Dataset's to have

Re: [VOTE] February PR of the Month

Gets my vote too! On 24 February 2023 22:26:55 GMT, Jarek Potiuk wrote: >Clearly #27758 > >On Fri, Feb 24, 2023 at 11:17 PM John Thomas > wrote: > >> Hi Folks, >> >> It's that time again! Vote for a PR that you think deserves it this month >> - our candidates from the script are as follows: >>

Re: [VOTE] Airflow Providers prepared on February 18, 2023

+1 binding On Feb 21 2023, at 3:06 pm, Josh Fell wrote: > +1 (non-binding) > > > On Tue, Feb 21, 2023 at 5:39 AM Pankaj Koti > wrote: > > +1 non-binding. Concurring with Rajath's tests. > > > > Regards, > > > > > > > > > > Pankaj Koti > > > > > > > > Senior Software Engineer, OSS Engineering

Re: [VOTE] Move K8S / Celery (and related) executors to respective providers

+1 binding On Feb 21 2023, at 9:21 am, Hussein Awala wrote: > > +1 non-binding.I'm a little concerned that this coupling will reduce the fast > evolution of providers, but given the benefits on the executor side, I vote > for it. > > From: Jarek Potiuk > Sent: Tuesday, February 21, 2023

Re: [Discussion] Set further policies for triaging issues

letting us know that and by having the proper >labels it might get more attraction to it. > > >On Sun, Feb 12, 2023 at 10:15 AM Ash Berlin-Taylor wrote: > >> I feel very strongly against automated closing of _issues_. >> >> There is nothing I find more infuriating

Re: [Discussion] Set further policies for triaging issues

I feel very strongly against automated closing of _issues_. There is nothing I find more infuriating and demoralising when dealing with an open source project (and big ones like Kubernetes are the worst offenders at this) where I find a bug or feature request is closed simply due to lack of

Re: [VOTE] AIP-53 OpenLineage in Airflow

+1 binding - I've been following the doc and comments and I think this will make Lineage a realistic possibility for all. -Ash On 10 February 2023 23:26:48 GMT, Julien Le Dem wrote: >Dear Airflow community, > >Following the discussion thread over the past few weeks, I'd like to call a >vote

[RESULT] AIP-52 Automatic setup and teardown tasks

This vote has passed with 20 +1 votes (13 binding, 7 non-binding) and no 0 or -1 votes. Binding votes: me Jarek Potiuk Elad Kal Josh Fell Ephraim Anierobi Ping Zhang Vikram Koka Pierre Jeambrun Felix Ullendall Kevin Yang Niko Oliveira Jed Cunningham Kaxil Naix Non-binding votes:

Re: [VOTE] AIP-52 Automatic setup and teardown tasks

t;setup/teardown operators - that would be a bit helpful, but there is no >such a thing. > >-- >,,,^..^,,, > > >On Mon, Jan 9, 2023 at 7:27 PM Ash Berlin-Taylor wrote: > >> Hello everyone, >> >> I am calling for a vote on AIP-52 >> https://cwiki.apache.org/conflu

Re: [VOTE][ISSUE-22073] Finalising approach for displaying non-ascii characters in DAG display name

ions. -ash On Jan 12 2023, at 9:50 pm, Ash Berlin-Taylor wrote: > > Possibly contentious idea: We allow unicode dag_ids for Postgres, MSQQL (and > sqllite) but for Mysql we enforce it as ASCII only. > On Jan 12 2023, at 6:15 pm, Jarek Potiuk wrote: > > As I mentioned multipl

Re: [VOTE][ISSUE-22073] Finalising approach for displaying non-ascii characters in DAG display name

code to dag_id (which currently > IMHO is not an option) my vote goes to 2. > > We can also drop MySQL support :D > > J. > > > > > On Thu, Jan 12, 2023 at 9:56 AM Ash Berlin-Taylor (mailto:a...@apache.org)> wrote: > > +1 to what Daniel said > > > > On 12 Jan

Re: [VOTE][ISSUE-22073] Finalising approach for displaying non-ascii characters in DAG display name

+1 to what Daniel said On 12 January 2023 08:32:29 GMT, Daniel Standish wrote: >1 appears to have potential fix: >https://github.com/apache/airflow/issues/21127#issuecomment-1030673862 >2. seems to fail due to our own ascii enforcement... what if we remove that? >3. does not appear to be

[VOTE] AIP-52 Automatic setup and teardown tasks

Hello everyone, I am calling for a vote on AIP-52 https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-52+Automatic+setup+and+teardown+tasks There haven't been any notable changes to the original document, mostly just clairifications to the proposal. This is my +1, and the vote will last

Re: [PROPOSAL] (slightly) change approach to default config values

+1 -- also been on my list of things I've thought about doing. (Get out of my head!) On Dec 20 2022, at 3:47 pm, Jarek Potiuk wrote: > Hello everyone, > > I have a proposal (Draft PR here: > https://github.com/apache/airflow/pull/28495) how we can change the approach > to default values for

Re: [Discuss]whether fail early for Xcom.set when value exceeds the BLOB limit

+1 -- though I don wonder if that is a behaviour that only MySQL exhibits? Either way, we should throw an exception early. -ash On Dec 20 2022, at 8:04 am, Jarek Potiuk wrote: > +1. I am all for failing it. I do not see an immediate issue with it - > especially if we also explain how to handle

[DISCUSS] AIP-52 Automatic setup and tear down tasks

Hi everyone, I'd like to start a discussion about a new feature we'd like to add to Airflow we call "setup and tear down tasks" https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-52+Automatic+setup+and+teardown+tasks Lets start with a code example, that if you are familiar with

Re: [PROPOSAL] (Internal) move of provider packages to isolated "providers" sub-folders

+7000 This sort of restructure has been on the idea list in the back of my head for at least a year if not two, but didn't ever crystlaize enough to do form it in to a concrete proposal. Love the prposed file structure, it was exactly what I had in mind. While it is verbose of having the path

Re: [VOTE] Airflow Providers prepared on December 14, 2022 (RC1)

+1 binding On Dec 16 2022, at 6:52 am, Phani Kumar wrote: > +1 (non-binding) . Our DAGs ran successfully. > > > On Thu, Dec 15, 2022 at 10:31 PM Jed Cunningham (mailto:jedcunning...@apache.org)> wrote: > > +1 (binding) > > > > Checked signatures, checksums, licences. > > Thanks for doing this

Re: [VOTE] New Provider: Cloudera

ame acceptable. > > Let me know if it is acceptable to break up these two discussions and have > > this vote move forward. > > Thank you, > > Regards. > > Philippe > > > > [1] https://lists.apache.org/thread/2z0lvgj466ksxxrbvofx41qvn03jrwwb > > [2] h

Re: [VOTE] New Provider: Cloudera

Just to break with the consensus: -1 Not because I don't think the provider would be useful or popular enough, precisely the opposite, and I'd like to see more companies maintain and manage their own providers and see an ecosystem of providers start to grow. Cloudera def has the means and

Re: [VOTE] Airflow Providers prepared on December 1st, 2022 (RC1)

+1 binding On 2 December 2022 10:00:19 GMT, Phani Kumar wrote: >+1 non-binding > >On Fri, Dec 2, 2022 at 3:12 PM Pankaj Koti > wrote: > >> +1 non-binding. >> >> Tested the Snowflake RC and our DAG ran fine with it. >> >> Regards, >> >> >> >> Pankaj Koti >> >> *Senior Software Engineer, *OSS

Re: [VOTE] Release Airflow 2.5.0 from 2.5.0rc3

+1 binding On Dec 1 2022, at 10:36 pm, Xiaodong Deng wrote: > +1 (binding) > > > Regards, > XD > > On Thu, Dec 1, 2022 at 14:26 Jarek Potiuk (mailto:ja...@potiuk.com)> wrote: > > +1 binding. Checked signatures, checksums, licences, and ran it via > > `breeze start airflow` with a number of

Re: [DISCUSS] Allow shorter voting periods for subsequent RCs

t release). On Nov 30 2022, at 9:57 am, Ephraim Anierobi wrote: > +1 to both (a) and (b) since RC2 up is usually a few commits > > On 2022/11/30 09:47:26 Ash Berlin-Taylor wrote: > > Hi All, > > > > We've just had a case where a 11th hour bug on 2.5.0rc2 (well technically, &g

[DISCUSS] Allow shorter voting periods for subsequent RCs

Hi All, We've just had a case where a 11th hour bug on 2.5.0rc2 (well technically, 12:01 as the vote time had finished, but we hadn't closed it yet/wouldn't have released anyway) https://github.com/apache/airflow/issues/28002. The fix is easy (it's a two line change, plus a bit of tidy up) but

Re: [VOTE] Airflow Providers prepared on November 26, 2022 (RC3)

+1 binding. On Nov 29 2022, at 8:01 am, Pankaj Koti wrote: > +1 non-binding (concurring with Phani) > > Regards, > > > > > Pankaj Koti > > > > Senior Software Engineer, OSS Engineering Team. > Location: Pune, India > > > Timezone: Indian Standard Time (IST) > > > Email:

Re: [VOTE] Release Airflow 2.5.0 from 2.5.0rc2

+1 binding. Tested a few dags, poked around the new dataset view (nice one Brent and Drew) - small improvements that'll make a big difference to usability. -ash On Nov 27 2022, at 6:50 am, Ephraim Anierobi wrote: > Hey fellow Airflowers, > > I have cut Airflow 2.5.0rc2. This email is calling a

Re: [VOTE] Release Airflow 2.5.0 from 2.5.0rc1

For minor releases there are too many changes, so we just use the thread. (Compare with patch releases where there should only be bug fixes to check) -a On 26 November 2022 11:46:14 GMT, Jeambrun Pierre wrote: >Hello all, > >Thanks Ephraim for preparing this new release. (This one packs a lot

Re: [AIP-44] PoC JSON RPC

Where did JSON-RPC come from? I'd like to catch up to speed on the discussion/reason. It is a very... crufty protocol and not one I would ever personally choose for new work. (Sorry I've been out a bit and may have missed some context or discussion.) -ash On 8 November 2022 15:53:06 GMT,

Re: [VOTE] Release Apache Airflow Helm Chart 1.7.0 based on 1.7.0rc1

+1 binding On Oct 10 2022, at 4:02 pm, Jed Cunningham wrote: > Hello Apache Airflow Community, > > This is a call for the vote to release Helm Chart version 1.7.0. > Consider this my (binding) +1. > The release candidate is available at: >

Re: [VOTE] Airflow Providers prepared on September 28, 2022

+1 binding On 30 September 2022 19:36:14 BST, Elad Kalif wrote: >+1 binding > >On Fri, Sep 30, 2022 at 12:36 AM Jed Cunningham >wrote: > >> +1 (binding) >> >> Checked signatures, checksums, and licences. >>

Re: [VOTE] Release Airflow 2.4.1 from 2.4.1rc1

+1 binding On 30 September 2022 19:55:08 BST, Josh Fell wrote: >+1 (non-binding) > >Ran a few DAGs and tested some bug fixes too. > >On Fri, Sep 30, 2022 at 2:21 PM Jed Cunningham >wrote: > >> Kind bump, I need 1 more vote on this release. >>

Airflow cncf-kubernetes provider 4.4.0 released

Hi everyone, Just a quick one this time - the apache-airflow-cncf-kubernetes provider v4.4.0 has been released to bring support for @task.kubernetes support! The source release, as well as the binary releases, are available here:

[RESULT] Release apache-airflow-providers-cncf-kubernetes from 4.4.0rc1

- checked signatures, hashes, licences. It installs cleanly on > > Airflow 2.4.0/ > > > > > > On Thu, Sep 22, 2022 at 5:32 PM Ash Berlin-Taylor > (mailto:a...@apache.org)> wrote: > > > A slightly unusal one this time (full provider release will happen next

[VOTE] Release apache-airflow-providers-cncf-kubernetes from 4.4.0rc

A slightly unusal one this time (full provider release will happen next week) but we want to get cncf.kubernetes out earlier to enable support for @task.kubernetes. This email is calling a vote on the release of apache-airflow-providers-cncf-kubernetes, which will last for 72 hours - which

Re: [LAZY CONSENSUS] enable string normalization on black

For clarity for those who didn't read the thread: This change will be applied just before 2.5 is going to be released (to avoid making cherry picks difficult) -a On 21 September 2022 18:03:19 BST, Daniel Standish wrote: >Proposal: remove param --skip-string-normalization on black pre-commit

Re: [DISCUSS] string normalization in black

> Happy to adapt to any style as long as I don't even have to think about it > and my personal preferences are not important here. That. On Sep 21 2022, at 12:45 pm, Jarek Potiuk wrote: > Yeah! Heated discussions! Tabs vs. Spaces. > > I am also full in for consistency. I think where there are

Re: [DISCUSS] AIP-50 Trigger DAG UI Extension with Flexible User Form Concept

Hey Jens, Nice idea. I've left a comment on the AIP page, suggesting that we use the existing DAG param typying (which is built on top of JSON schema) -- to me it has a number of advtanges a) easier to secure (declarative, rather than executing code) which is a good goal as it means it can

Re: CVE-2022-38054: Apache Airflow: Session Fixation

We would also like to thank Ali Al-Habsi for independently discovering and reporting this issue. Thanks, Ash, Apache Airflow PMC On 2022/09/02 03:53:55 Jedidiah Cunningham wrote: > Description: > > In Apache Airflow versions 2.2.4 through 2.3.3, the `database` webserver > session backend was

Re: [DISCUSS] Migrate airflow code to use PEP 563 (postponed evaluation of annotations) MAYBE?

ully to v2-4 > branch, making sure that I do not drag-along unwanted changes from main. > > TP - I hope it all (the discussion and proposal) alleviates your concerns? > > J. > On Mon, Sep 12, 2022 at 11:10 AM Ash Berlin-Taylor (mailto:a...@apache.org)> wrote: > > Changes a

Re: [DISCUSS] Migrate airflow code to use PEP 563 (postponed evaluation of annotations) MAYBE?

for core airflow, and one for providers, say. I don't feel that strongly about this, it's just how I would have done it personally -- and having a single rev lets me more easily ignore that rev. -ash On Sun, Sep 11 2022 at 14:29:56 +01:00:00, Ash Berlin-Taylor wrote: Right okay, gotcha. I'll

Re: [DISCUSS] Migrate airflow code to use PEP 563 (postponed evaluation of annotations) MAYBE?

>merged code while cherry-picking). > >This was the reasoning behind doing it fast when I recalled that we hold >off from it in the middle of 2.3 development. > >J,. > > >On Sun, Sep 11, 2022 at 1:19 PM Ash Berlin-Taylor wrote: > >> One comment on the timing: but

Re: [DISCUSS] Migrate airflow code to use PEP 563 (postponed evaluation of annotations) MAYBE?

One comment on the timing: but doing it now you make cherry picks for 2.4.x harder, of which were about to do a lot of. The best time to do such a change is just _before_ cutting the next main branch. On 10 September 2022 21:38:12 BST, Jarek Potiuk wrote: >Hey Everyone, > >TL;DR; I planned to

Airflow Providers released on September 08, 2022 are ready

Dear Airflow community, I'm happy to announce that new versions of Airflow Providers packages were just released.

[RESULT] Airflow Providers prepared on 2022-09-05

Hello, Apache Airflow Providers (based on RC1) have been accepted. 3 "+1" binding votes received: - Ash Berlin-Taylor - Kaxil Naik - Elad Kalif and 3 "+1+ non-binding votes: - Pankaj Koti - Phani Kumar - Pankaj Singh Thanks for voting everyone. The release will happen shortly.

Apache Airflow 2.4.0b1 available for testing

Hello everyone! I'm very excited to let you all know that I have just push 2.4.0b1 to PyPi (docker images coming soon) and is now ready for testing. All going well we will have an RC1 next week. The headline user-facing feature is AIP-48: Data-aware scheduling

Re: [VOTE] Airflow Providers prepared on 2022-09-05

:faceplam:. For clarity the vote will end on Thursday, 2022-09-08 at 15:56 UTC. On Mon, Sep 5 2022 at 16:56:42 +01:00:00, Ash Berlin-Taylor wrote: Hey all, I have just cut the new wave Airflow Providers packages. This email is calling a vote on the release, which will last for 72 hours

Re: Airflow Pluggable Scheduler

Yeah, essentially what `airflow scheduler` does is: ``` job = SchedulerJob() job.run() ``` You could even put `if __name__ == "__main__": ...` at the end of your custom scheduler and then do `python -m your.custom_scheduler` On Tue, Sep 6 2022 at 12:15:52 +02:00:00, Jarek Potiuk wrote:

[VOTE] Airflow Providers prepared on 2022-09-05

Hey all, I have just cut the new wave Airflow Providers packages. This email is calling a vote on the release, which will last for 72 hours - which means that it will end on $(date -d '+3 days'). Consider this my (binding) +1. Notable changes here (worthy of an out-of-band fix) - Fix

Re: [DISCUSS] "Use existing venv" support for PythonVirtualenvOperator as counterpart to AIP-46

e today and they said they would >>>>>>> ABSOLUTELY love it. They were actually blocked from migrating to 2.3.3 >>>>>>> because one of their teams needed a DBT environment while the other >>>>>>> team needed some other dependency and they ar

Re: Should value passed via XComArg be templated?

In this example I'd say it shouldn't be templated, as you can do it via @task def get_templated_echo(ds): return f'echo "The date is {ds}"' (This is nothing particular to XcomArg either, just the way Xcom had always worked.) I don't think we can change the current/default behaviour without

Re: [VOTE] Release Airflow 2.3.4 from 2.3.4rc1

+1 (binding) On 20 August 2022 15:25:24 BST, Jarek Potiuk wrote: >+1 (binding). > >Tested all my changes, run a few dags, verified licences, signatures, >checksums. Also verified refreshed dependencies and constraints. All looks >good. > >On Sat, Aug 20, 2022 at 4:08 PM Jeambrun Pierre >wrote:

Re: [DISCUSS] "Use existing venv" support for PythonVirtualenvOperator as counterpart to AIP-46

gt; On 13 Aug 2022, at 02:52, Jeambrun Pierre wrote: >> >> I feel like this is a great alternative at the price of a very moderate >> effort. (I'd be glad to help with it). >> >> Mutually exclusive sounds good to me as well. >> >> Best, >> Pierre >> >> Le

Re: User Kinceo1 posting spam review approvals

We'll need to open a jira ticket in the INFRA queue similar to (As a commiter you can log in to that Jira instance with your asf credentials I think) Link to all their interactions which is probably useful to include

CVE-2022-38362: Apache Airflow Docker Provider <3.0 RCE vulnerability in example dag

Description: Apache Airflow Docker's Provider shipped with an example DAG that was vulnerable to (authenticated) remote code exploit of code on the Airflow worker host. Mitigation: Disable loading of example DAGs or upgrade the apache-airflow-providers-docker to 3.0.0 or above Credit:

Re: [DISCUSS] "Use existing venv" support for PythonVirtualenvOperator as counterpart to AIP-46

Yes, this has been on my background idea list for an age -- I'd love to see it happen! Have you thought about how it would behave when you specify an existing virtualenv and include requirements in the operator that are not already installed there? Or would they be mutually exclusive? (I

  1   2   3   4   5   6   7   8   >