+1 to this idea! Best, Wei
> On Feb 13, 2025, at 5:29 AM, Vikram Koka <vik...@astronomer.io.invalid> wrote: > > Hey everyone, > > I created the following (very early) draft PR > <https://github.com/apache/airflow/pull/46694> to introduce and socialize > the concept of a "Common Message Queue" abstraction, similar to the "Common > SQL" and "Common IO" abstractions already present in Airflow. > > This "Common Message Queue" abstraction will be a provider package similar > to the other "Common" packages and is intended to be a simple abstraction > over Apache Kafka, Amazon SQS, Google PubSub, and others messaging > frameworks. > > One of the key initial goals is to provide a simple DAG writing abstraction > for integrating Event Driven Scheduling coming with Airflow 3 with > streaming systems such as Kafka. This proposal was crafted in collaboration > with Vincent Beck, who has been diligently working through AIP-82 as part > of Airflow 3. > > At this stage, this is very much a work in progress draft and is intended > to solicit feedback, rather than in any way a completed incarnation. > > Please feel free to comment on the PR directly or reach out to me over > email or slack. > > Best regards, > Vikram > > -- > > Vikram Koka > Chief Strategy Officer > Email: vik...@astronomer.io > > > <https://www.astronomer.io/> --------------------------------------------------------------------- To unsubscribe, e-mail: dev-unsubscr...@airflow.apache.org For additional commands, e-mail: dev-h...@airflow.apache.org
