One thought that occurred to me... In order to really take advantage of
async python, it may essentially require a scheduler rewrite. Certainly
there will be a lot of refactoring required. And I have not looked closely
at this. But I think chances are good that there will be a lot of things
that
: Hussein Awala
Sent: Wednesday, April 10, 2024 5:44:48 PM
To: dev@airflow.apache.org
Subject: Re: [DISCUSS] Asynchronous SQLAlchemy
What you propose is exactly what `sync_to_async` does implicitly (check source
code
<https://github.com/django/asgiref/blob/db3ff43e9fa1daf7c6b1cb67db8eec1885be9
ere
> that helped.
>
> I imagine this approach would work in a lot of cases where only a
> synchronous driver can be used but you're working in an asynchronous
> context, hope this idea is helpful.
>
> Damian
>
>
> -Original Message-
> From: Amogh Desai
&g
ing in an asynchronous context, hope this
idea is helpful.
Damian
-Original Message-
From: Amogh Desai
Sent: Wednesday, April 10, 2024 12:33 AM
To: dev@airflow.apache.org
Subject: Re: [DISCUSS] Asynchronous SQLAlchemy
Thanks, Hussein and Ash, I am much clearer on the scope now and I am
t; daniel.stand...@astronomer.io> wrote:
> > > >
> > > > > https://github.com/omnilib/aiosqlite maybe?
> > > > >
> > > > > On Mon, Apr 8, 2024 at 1:03 PM Scheffler Jens (XC-AS/EAE-ADA-T)
> > > > > wrote:
> &
gt;
> > > > https://github.com/omnilib/aiosqlite maybe?
> > > >
> > > > On Mon, Apr 8, 2024 at 1:03 PM Scheffler Jens (XC-AS/EAE-ADA-T)
> > > > wrote:
> > > >
> > > >> I understand the „all-in“ approach as we were dropping MSSQL… how
> can
&
; > > > wrote:
> > > >
> > > >> I understand the „all-in“ approach as we were dropping MSSQL… how
> can
> > we
> > > >> keep one codenase bit cooe with sqlite? I assume we must support
> this
> > at
> > >
MSSQL… how can
> we
> > >> keep one codenase bit cooe with sqlite? I assume we must support this
> at
> > >> least for dev setups.
> > >>
> > >> Sent from Outlook for iOS<https://aka.ms/o0ukef>
> > >>
> >> From: Jarek Potiuk
> >> Sent: Monday, April 8, 2024 8:30:18 PM
> >> To: dev@airflow.apache.org
> >> Subject: Re: [DISCUSS] Asynchronous SQLAlchemy
> >>
> >> Yep. If we can make both Postgres and MySQL work with Async - I am also
> >>
OS<https://aka.ms/o0ukef>
> >>
> >> From: Jarek Potiuk
> >> Sent: Monday, April 8, 2024 8:30:18 PM
> >> To: dev@airflow.apache.org
> >> Subject: Re: [DISCUSS] Asynchronous SQLAlchemy
> >>
> >&
ev setups.
>
> Sent from Outlook for iOS<https://aka.ms/o0ukef>
>
> From: Jarek Potiuk
> Sent: Monday, April 8, 2024 8:30:18 PM
> To: dev@airflow.apache.org
> Subject: Re: [DISCUSS] Asynchronous SQLAlchemy
>
> Yep. If we can make both Post
___
>> From: Jarek Potiuk
>> Sent: Monday, April 8, 2024 8:30:18 PM
>> To: dev@airflow.apache.org
>> Subject: Re: [DISCUSS] Asynchronous SQLAlchemy
>>
>> Yep. If we can make both Postgres and MySQL work with Async - I am also
>> all
>> for the &
pril 8, 2024 8:30:18 PM
To: dev@airflow.apache.org
Subject: Re: [DISCUSS] Asynchronous SQLAlchemy
Yep. If we can make both Postgres and MySQL work with Async - I am also all
for the "All" approach. If it means that we need to support only certain
drivers and certain versions
Yep. If we can make both Postgres and MySQL work with Async - I am also all
for the "All" approach. If it means that we need to support only certain
drivers and certain versions of the DBs - so be it. As mentioned in my
original comments (long time ago when we had MSSQL support) - this was not
I wholeheartedly agree with Ash that it should be all or nothing. And
*all* sounds
better to me :)
On Mon, Apr 8, 2024 at 10:54 AM Ash Berlin-Taylor wrote:
> I’m all in favour of async SQLAlchemy. We’ve built two products
> exclusively at @ Astronomer that use sqlalchemy+psycopg3+async and
I’m all in favour of async SQLAlchemy. We’ve built two products exclusively at
@ Astronomer that use sqlalchemy+psycopg3+async and love it. Async does take a
bit of a learning curve, but SQLA has done it nicely and it works really well.
I think this needs to be an all or nothing thing — having
I checked the content and the PR that you attached.
The results do seem promising and I like the general idea of this approach.
But as Jarek
also mentioned on the PR:
1. Not everyone might be on the board to go all async due to certain
limitations around
access to the drivers, or corporate
The Metadata Database is the brain of Airflow, where all scheduling
decisions, cross-communication, synchronization between components, and
management via the web server, are made using this database.
One option to optimize the DB queries is to merge many into a single query
to reduce latency and
18 matches
Mail list logo
