+1 to 3. For cases where absolute minimal latency is critical, and worker resources aren't constrained, poke mode could still be the optimal choice. I don't see any value in reschedule mode anymore, deferrable should be the default.
On Wed, Nov 13, 2024 at 12:21 PM Kaxil Naik <kaxiln...@gmail.com> wrote: > There is 4th option to keep things as-is too :) > > On Wed, 13 Nov 2024 at 12:19, Kaxil Naik <kaxiln...@gmail.com> wrote: > > > Hi all, > > > > Following up on the Dev call discussions last Thursday, I am opening this > > up for discussion. > > > > Reschedule mode was introduced to improve efficiency over poke mode by > > allowing tasks to wait without holding a worker slot. Since the > > introduction of deferrable operators in Airflow 2.2, however, we now have > > an even more optimal, async-driven solution. The adoption of deferrable > > operators has been really good, and since we are already chopping things > > off with Airflow 3 it might be time to consider making them the default > > mode. > > > > This will ensure that our users always have the most optimal way of > > running sensors by default and that we, the maintainers or folks > supporting > > Airflow deployments in companies, do not need to know different > approaches > > with Reschedule mode, either. > > > > However, not all sensors can be async, either due to limitations in > > underlying libraries or a lack of unique ids for async polling. > > > > Knowing that we have a few options: > > > > 1) *Remove Poke & Reschedule modes* > > > > This is aggressive and it means we will have to remove things like > > PostgresSensor that does not support async. > > > > 2) *Remove Reschedule mode * > > > > Make deferrable the primary mode, falling back to poke where async isn’t > > supported. > > > > 3) *Make Deferrable the default, keep Poke & Reschedule* > > > > This is a defensive option that maintains current behaviour but ensures > > that we have the most performant option by default. It could be as simple > > as making AIRFLOW__OPERATORS__DEFAULT_DEFERRABLE default to True. > > > > I’d love to hear feedback, especially from users who rely on reschedule > > mode today! > > > > Regards, > > Kaxil > > >
