Hi Wei Lee, Thanks for the review, While I was working on the POC , I had a bit of confusion about how to use the logic present inside the sensor execute method. for both with and without triggerer flow. so to make it work for both flows, I have moved out to execute logic with two methods.
I appreciate the idea of a simple helper function, I'm eager to hear any suggestions you might have or anyone :). Regards, Pavan Kumar On Wed, Aug 14, 2024 at 12:16 PM Wei Lee <weilee...@gmail.com> wrote: > Thank you for bringing this up! I have added some comments to the > document. I'm unsure if we really want or need to implement more complex > logic for this. What I have in mind is simply adding helper functions to > InternalSensorTrigger and continuing to use the run method in BaseTrigger. > The main purpose of those methods is to allow operator authors to change > the sensors to leverage this start/end from trigger more easily. > > Best, > Wei > > > On Aug 14, 2024, at 5:19 PM, Pavankumar Gopidesu < > gopidesupa...@gmail.com> wrote: > > > > Hi Jarek, > > > > Thanks for the questions :) , > > > > I completely agree with you , from 2.10 we have the start_from_trigger > > parameter, which , when set , > > allows a task to be executed directly from the triggerrer without worker > > involvement. I believe that for > > any sensor to be executed in the triggerer, a corresponding trigger > > implementation must be in place. > > Good thing is , most of the required trigger implementations are already > > available in current airflow providers. > > > > Additionally, i agree that there is no real difference between a > deferrable > > sensor and a deferrable operator, > > generally , users utilize the BaseSensorOperator to create custom > sensors, > > However if they want to run > > these sensors in triggerer, they must implement the triggerer's run > method > > by extending the base trigger class. > > > > The key difference with this proposal is the introduction of a common > > trigger implementation (referred to in this > > POC as InternalSensorTrigger )[1]. This would allow users to use the > > new feature start_from_trigger param > > with their custom sensor and eliminates the need for individual trigger > > implementations for each custom sensor > > when they create. > > > > Alternatively, we could leave the start_from_trigger parameter in > > BaseSensorOperator with a default > > value as false and let users decide whether they want to run the sensor > in > > triggerrer. If they choose to do so, they > > can simply set the parameter to true and triggerrer uses the > > InternalSensorTrigger class. > > > > Apologies if my answer is too confusing :) > > > > [1]: > > > https://github.com/apache/airflow/pull/41355/files#diff-7486f32e385d7ad0376cccda08d80e54939aa901a24616d11fb1f5cba6af7f83R144 > > > > Regards, > > Pavan Kumar > > > > On Wed, Aug 14, 2024 at 12:29 AM Jarek Potiuk <ja...@potiuk.com> wrote: > > > >> How does it differ from the upcoming 2.10 feature? > >> > >> * Deferrable operators can now execute directly from the triggerer > without > >> needing to go through the worker. This is especially efficient for > certain > >> operators, like sensors, and can help teams save both time and money. > >> > >> As of 2.10 - Sensors already can run mostly in Triggerrer and basically > >> there is no big difference any more between deferrable sensor and > >> deferrable operator. > >> > >> > >> On Mon, Aug 12, 2024 at 3:59 AM Kaxil Naik <kaxiln...@gmail.com> wrote: > >> > >>> Thanks for putting this together, I will take a look this week. > >>> > >>> On Fri, 9 Aug 2024 at 13:12, Pavankumar Gopidesu < > >> gopidesupa...@gmail.com> > >>> wrote: > >>> > >>>> Hi All, > >>>> > >>>> I have created a draft document for Sensor Improvements using > triggers. > >>>> > >>>> Details: > >>>> > >>>> > >>> > >> > https://docs.google.com/document/d/1Kb_wL-T1DHkOpmR_QNa3O5p_2hMTLzM-sb_HzQ0PYCo/edit?usp=sharing > >>>> > >>>> POC: > >>>> https://github.com/apache/airflow/pull/41355 > >>>> > >>>> This is my first draft post, apologies if any mistakes in the > >> document. I > >>>> would greatly appreciate your insights and suggestions on draft. > >>>> > >>>> Thank you very much for your time. > >>>> > >>>> Regards, > >>>> Pavan > >>>> > >>> > >> > > > --------------------------------------------------------------------- > To unsubscribe, e-mail: dev-unsubscr...@airflow.apache.org > For additional commands, e-mail: dev-h...@airflow.apache.org > >
