Definitely +1, AIP-85 can also greatly benefit from this, e.g. for event-based DAG processing
On Fri, Feb 14, 2025 at 12:31 AM Vikram Koka <vik...@astronomer.io.invalid> wrote: > Thanks for the quick feedback, team! > > Shubham, > In response to your question about Notifier, > I was in two minds about it, but I did include the > "MsgQueuePublishOperator" as a specific Operator for that particular > reason. > > I will update the example to be explicit. > > > > > > On Thu, Feb 13, 2025 at 3:17 PM Jens Scheffler <j_scheff...@gmx.de.invalid > > > wrote: > > > +1 and looking forward that I have time to read the PR... (not today... > > but the next days!) > > > > On 13.02.25 08:09, Jarek Potiuk wrote: > > > +1 for sure. This is a great idea. > > > > > > On Thu, Feb 13, 2025 at 5:40 AM Ben Tallman <btall...@gmail.com> > wrote: > > > > > >> +1 for sure! > > >> > > >> Thanks, > > >> Ben > > >> > > >> -- > > >> Ben Tallman - 503.680.5709 <(503)%20680-5709> > > >> > > >> > > >> On Wed, Feb 12, 2025 at 8:19 PM Abhishek Bhakat > > >> <abhishek.bha...@astronomer.io.invalid> wrote: > > >> > > >>> +1 yay! event driven scheduling soon > > >>> > > >>> Regards, > > >>> Avi > > >>> > > >>> On Thu, Feb 13, 2025 at 2:02 AM Wei Lee <weilee...@gmail.com> wrote: > > >>> > > >>>> +1 to this idea! > > >>>> > > >>>> Best, > > >>>> Wei > > >>>> > > >>>>> On Feb 13, 2025, at 5:29 AM, Vikram Koka > > >> <vik...@astronomer.io.invalid > > >>>> wrote: > > >>>>> Hey everyone, > > >>>>> > > >>>>> I created the following (very early) draft PR > > >>>>> <https://github.com/apache/airflow/pull/46694> to introduce and > > >>>> socialize > > >>>>> the concept of a "Common Message Queue" abstraction, similar to the > > >>>> "Common > > >>>>> SQL" and "Common IO" abstractions already present in Airflow. > > >>>>> > > >>>>> This "Common Message Queue" abstraction will be a provider package > > >>>> similar > > >>>>> to the other "Common" packages and is intended to be a simple > > >>> abstraction > > >>>>> over Apache Kafka, Amazon SQS, Google PubSub, and others messaging > > >>>>> frameworks. > > >>>>> > > >>>>> One of the key initial goals is to provide a simple DAG writing > > >>>> abstraction > > >>>>> for integrating Event Driven Scheduling coming with Airflow 3 with > > >>>>> streaming systems such as Kafka. This proposal was crafted in > > >>>> collaboration > > >>>>> with Vincent Beck, who has been diligently working through AIP-82 > as > > >>> part > > >>>>> of Airflow 3. > > >>>>> > > >>>>> At this stage, this is very much a work in progress draft and is > > >>> intended > > >>>>> to solicit feedback, rather than in any way a completed > incarnation. > > >>>>> > > >>>>> Please feel free to comment on the PR directly or reach out to me > > >> over > > >>>>> email or slack. > > >>>>> > > >>>>> Best regards, > > >>>>> Vikram > > >>>>> > > >>>>> -- > > >>>>> > > >>>>> Vikram Koka > > >>>>> Chief Strategy Officer > > >>>>> Email: vik...@astronomer.io > > >>>>> > > >>>>> > > >>>>> <https://www.astronomer.io/> > > >>>> > > >>>> > --------------------------------------------------------------------- > > >>>> To unsubscribe, e-mail: dev-unsubscr...@airflow.apache.org > > >>>> For additional commands, e-mail: dev-h...@airflow.apache.org > > >>>> > > >>>> > > > > --------------------------------------------------------------------- > > To unsubscribe, e-mail: dev-unsubscr...@airflow.apache.org > > For additional commands, e-mail: dev-h...@airflow.apache.org > > > > >
