+1, looking forward to it. Regards, Rahul
On Thu, 13 Feb, 2025, 03:00 Vikram Koka, <vik...@astronomer.io.invalid> wrote: > Hey everyone, > > I created the following (very early) draft PR > <https://github.com/apache/airflow/pull/46694> to introduce and socialize > the concept of a "Common Message Queue" abstraction, similar to the "Common > SQL" and "Common IO" abstractions already present in Airflow. > > This "Common Message Queue" abstraction will be a provider package similar > to the other "Common" packages and is intended to be a simple abstraction > over Apache Kafka, Amazon SQS, Google PubSub, and others messaging > frameworks. > > One of the key initial goals is to provide a simple DAG writing abstraction > for integrating Event Driven Scheduling coming with Airflow 3 with > streaming systems such as Kafka. This proposal was crafted in collaboration > with Vincent Beck, who has been diligently working through AIP-82 as part > of Airflow 3. > > At this stage, this is very much a work in progress draft and is intended > to solicit feedback, rather than in any way a completed incarnation. > > Please feel free to comment on the PR directly or reach out to me over > email or slack. > > Best regards, > Vikram > > -- > > Vikram Koka > Chief Strategy Officer > Email: vik...@astronomer.io > > > <https://www.astronomer.io/> >
