One use case that does not work with defer is when you're checking resources that are only available on a subset of Airflow workers.
While my team produces reports for the entire company, we must segregate different business team's information, and to ensure this there is a policy that no server has access to information from both teams. We separate our workers and set certain Airflow tasks to one server or the other, for this case we cannot set sensors to deferrable, because the Airflow defer process does not support being able to choose which triggerer to send to. I am sure that we aren't the only company with this problem and are using Airflow workers in this way. Damian -----Original Message----- From: ambika garg <ambikagarg1...@gmail.com> Sent: Wednesday, November 13, 2024 11:01 AM To: dev@airflow.apache.org Subject: Re: [DISCUSSION] Replace Poke & Reschedule mode from Sensors for Airflow 3 in favor of Deferrable I would vote for option 3 as well, making deferrable operators the default mode ensures that users benefit from the most efficient, async-driven solution without requiring any additional configuration changes. Also, keeping Poke and Reschedule modes ensures backward compatibility with existing operators and users who rely on these modes. On Wed, Nov 13, 2024 at 9:58 AM Vincent Beck <vincb...@apache.org> wrote: > I am definitely in favor of considering the deferrable mode as the > default one. Between 1 and 3, even though I am a big fan of removing > and simplifying things in general, I feel like (no real data here) we > are not ready for 1 yet. So my vote would go to 3. I feel like > removing the poke mode would require too much work on the operators. > > On 2024/11/13 13:59:35 Jarek Potiuk wrote: > > I am torn between 1) and 3). While 1) is tempting and it would > > simplify > our > > state management, I think 3) is the safer choice. I think if we were > > not able to convert all our operators to be deferrable yet, there > > are > probably > > many thousands of custom ones that will stop working if we remove > > that feature. > > > > If we go 1) but then any operator that would go to "poke & > > reschedule", should just be "normal sensor" and simply start taking > > resources while waiting. Technically it's not "breaking" the flow, > > but it's likely > breaking > > installation which heavily relies on rescheduling and it would > dramatically > > increase resource usage. And there is no easy way out short of > > rewriting all such operators to support deferrable. > > > > I think personally that in making such decision we should consider > > two > > things: > > > > 1) will this stop some people from migrating to Airflow 3 because it > > will be "heavy operation"? > > 2) how likely we think it's going to happen - will we use big, > > important users who might be "success" stories for Airflow 3. > > > > I have no data to back it up, (maybe some people here could have it) > > - > but > > my intuition tells me that: > > > > 1) yes it will stop some users from migrating to Airflow 3 (because > > they will either have to accept increased resource usage or find > > engineering time to rewrite their custom operators) > > 2) yeah, I think it's quite likely and quite likely big users that > > could > be > > "Airflow 3 success story" might be affected > > > > But I am guessing. If someone could provide some data telling that > > either > > 1) or 2) assumption I made is false, I am happy to support option > > 1). For now it's 3). > > > > > > > > On Wed, Nov 13, 2024 at 2:43 PM Abhishek Bhakat > > <abhishek.bha...@astronomer.io.invalid> wrote: > > > > > +1 to 3. For cases where absolute minimal latency is critical, and > worker > > > resources aren't constrained, poke mode could still be the optimal > choice. > > > I don't see any value in reschedule mode anymore, deferrable > > > should be > the > > > default. > > > > > > On Wed, Nov 13, 2024 at 12:21 PM Kaxil Naik <kaxiln...@gmail.com> > wrote: > > > > > > > There is 4th option to keep things as-is too :) > > > > > > > > On Wed, 13 Nov 2024 at 12:19, Kaxil Naik <kaxiln...@gmail.com> > wrote: > > > > > > > > > Hi all, > > > > > > > > > > Following up on the Dev call discussions last Thursday, I am > opening > > > this > > > > > up for discussion. > > > > > > > > > > Reschedule mode was introduced to improve efficiency over poke > mode by > > > > > allowing tasks to wait without holding a worker slot. Since > > > > > the introduction of deferrable operators in Airflow 2.2, > > > > > however, we > now > > > have > > > > > an even more optimal, async-driven solution. The adoption of > deferrable > > > > > operators has been really good, and since we are already > > > > > chopping > > > things > > > > > off with Airflow 3 it might be time to consider making them > > > > > the > default > > > > > mode. > > > > > > > > > > This will ensure that our users always have the most optimal > > > > > way of running sensors by default and that we, the maintainers > > > > > or folks > > > > supporting > > > > > Airflow deployments in companies, do not need to know > > > > > different > > > > approaches > > > > > with Reschedule mode, either. > > > > > > > > > > However, not all sensors can be async, either due to > > > > > limitations in underlying libraries or a lack of unique ids for async > > > > > polling. > > > > > > > > > > Knowing that we have a few options: > > > > > > > > > > 1) *Remove Poke & Reschedule modes* > > > > > > > > > > This is aggressive and it means we will have to remove things > > > > > like PostgresSensor that does not support async. > > > > > > > > > > 2) *Remove Reschedule mode * > > > > > > > > > > Make deferrable the primary mode, falling back to poke where > > > > > async > > > isn’t > > > > > supported. > > > > > > > > > > 3) *Make Deferrable the default, keep Poke & Reschedule* > > > > > > > > > > This is a defensive option that maintains current behaviour > > > > > but > ensures > > > > > that we have the most performant option by default. It could > > > > > be as > > > simple > > > > > as making AIRFLOW__OPERATORS__DEFAULT_DEFERRABLE default to True. > > > > > > > > > > I’d love to hear feedback, especially from users who rely on > reschedule > > > > > mode today! > > > > > > > > > > Regards, > > > > > Kaxil > > > > > > > > > > > > > > > > --------------------------------------------------------------------- > To unsubscribe, e-mail: dev-unsubscr...@airflow.apache.org > For additional commands, e-mail: dev-h...@airflow.apache.org > > ________________________________ Strike Technologies, LLC (“Strike”) is part of the GTS family of companies. Strike is a technology solutions provider, and is not a broker or dealer and does not transact any securities related business directly whatsoever. This communication is the property of Strike and its affiliates, and does not constitute an offer to sell or the solicitation of an offer to buy any security in any jurisdiction. It is intended only for the person to whom it is addressed and may contain information that is privileged, confidential, or otherwise protected from disclosure. Distribution or copying of this communication, or the information contained herein, by anyone other than the intended recipient is prohibited. If you have received this communication in error, please immediately notify Strike at i...@striketechnologies.com, and delete and destroy any copies hereof. ________________________________ CONFIDENTIALITY / PRIVILEGE NOTICE: This transmission and any attachments are intended solely for the addressee. This transmission is covered by the Electronic Communications Privacy Act, 18 U.S.C ''2510-2521. The information contained in this transmission is confidential in nature and protected from further use or disclosure under U.S. Pub. L. 106-102, 113 U.S. Stat. 1338 (1999), and may be subject to attorney-client or other legal privilege. Your use or disclosure of this information for any purpose other than that intended by its transmittal is strictly prohibited, and may subject you to fines and/or penalties under federal and state law. If you are not the intended recipient of this transmission, please DESTROY ALL COPIES RECEIVED and confirm destruction to the sender via return transmittal. --------------------------------------------------------------------- To unsubscribe, e-mail: dev-unsubscr...@airflow.apache.org For additional commands, e-mail: dev-h...@airflow.apache.org
