Re: [HUGE DISCUSSION] Airflow3 and tactical (Airflow 2) vs strategic (Airflow 3) approach

Good point Jed. I responded back to your comment in the doc as well and very open to changing the term in the doc. Used the term "interactive DAG run" as the ability to invoke or trigger a DAG run through the API, with the expectation of getting back a result immediately. An alternate term could

Re: [HUGE DISCUSSION] Airflow3 and tactical (Airflow 2) vs strategic (Airflow 3) approach

Very exciting! Looks like we will have a busy period of time ahead of us. Overall I like the plan so far, especially using this year's Airflow Summit as an opportunity to announce and gather feedback, and the 2025 version to pitch upgrading. I left a comment in the doc, but we might want to

Re: [HUGE DISCUSSION] Airflow3 and tactical (Airflow 2) vs strategic (Airflow 3) approach

Hi all, As promised, we are pleased to share our proposal for Airflow 3 . We met several community members in the last few days to get feedback on this proposal, and we are glad to say that most of

Re: Refactor Scheduler Timed Events to be Async?

One thing to bear in mind here is the number of db connections - each connection can only be used by one thread or coroutine at a time, so even when the scheduler is changed to use async db calls, we might not be able to do a lot of the scheduled tasks concurrently. At least not without

Re: Refactor Scheduler Timed Events to be Async?

But you could run them in a thread or subprocess. Another option would be to just take all of the timed events and make them all asyncio and then run them all via asyncio in one continually running thread. That would be a bite size step towards AIP-70. Though, it might be a large bite :) On

Re: [DISCUSS] simplifying try_number handling

+1 Pumped to remove confusion around tries On Fri, May 3, 2024 at 5:01 AM Wei Lee wrote: > Thanks, Daniel! +1 for this one. This was confusing when I worked on the > starting from triggerer stuff. > > Best, > Wei > > > > On May 3, 2024, at 11:59 AM, Amogh Desai > wrote: > > > > Looks good to

Re: Refactor Scheduler Timed Events to be Async?

If we don't have many Asyncio tasks running in the event loop, there will not be any benefit from migrating to asynchronous, IMHO it will be anyway rewritten to be asynchronous as a part of AIP-70

Refactor Scheduler Timed Events to be Async?

This might be a dumb question as I don't have experience with asyncio, but should the EventScheduler in the Airflow scheduler be rewritten to be asynchronous? The so called "timed events" (e.g. zombie reaping,

Re: [VOTE] Airflow Providers prepared on May 01, 2024

+1 (non-binding) Tested my set of changes. Best regards, *Pankaj Koti* Senior Software Engineer (Airflow OSS Engineering team) Location: Pune, Maharashtra, India Timezone: Indian Standard Time (IST) On Fri, May 3, 2024 at 4:25 PM Hussein Awala wrote: > +1 (binding) checked licences,

Re: [VOTE] Airflow Providers prepared on May 01, 2024

+1 (binding) checked licences, checksums, signatures and sources, and ran some testing dags for cncf.kuberenetes and amazon providers. On Fri, May 3, 2024 at 9:29 AM Wei Lee wrote: > +1 (non-binding) > > Best, > Wei > > > On May 3, 2024, at 2:53 PM, Pankaj Singh > wrote: > > > > +1

Re: [DISCUSS] simplifying try_number handling

Thanks, Daniel! +1 for this one. This was confusing when I worked on the starting from triggerer stuff. Best, Wei > On May 3, 2024, at 11:59 AM, Amogh Desai wrote: > > Looks good to me. > > Personally I never ran into any issues with this so far but I agree with > the issues it solves. >

Re: [VOTE] Airflow Providers prepared on May 01, 2024

+1 (non-binding) Best, Wei > On May 3, 2024, at 2:53 PM, Pankaj Singh wrote: > > +1 (non-binding) > > On Thu, May 2, 2024 at 6:03 PM Elad Kalif wrote: > >> I will exclude pinecone from this release. >> Please continue voting excluding pinecone provider >> >> On Thu, May 2, 2024 at 3:19 PM

Re: [VOTE] Airflow Providers prepared on May 01, 2024

+1 (non-binding) On Thu, May 2, 2024 at 6:03 PM Elad Kalif wrote: > I will exclude pinecone from this release. > Please continue voting excluding pinecone provider > > On Thu, May 2, 2024 at 3:19 PM Ankit Chaurasia > wrote: > > > -1 non-binding for Pinecone: 2.0.0rc1 > > > > There is an issue