Hi all, After a period of designing and prototyping, I've completed a first draft of AIP-40 that I'd love to get some feedback on from the community: https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=177050929
This AIP proposes a way of adding what I'm calling "deferrable" Operators into Airflow - essentially, taking the goal of Smart Sensors and making it much more generalisable, where any Sensor or Operator can choose to "defer" its execution based on an asynchronous trigger, and where all the triggers run in one (or more) processes for efficiency. It also means that any Operator can be made deferrable in a backwards-compatible way should we wish to in future, though I'm not proposing that for a first release. It comes with a working prototype, too, should you wish to see what kind of code footprint this would have: https://github.com/apache/airflow/pull/15389 I personally think this would be a huge improvement for Airflow's efficiency - I suspect we might be able to reduce the amount of resources some Airflow installs use by over 50% if all their idling operators were ported to be deferrable - but I would welcome further opinions. Thanks, Andrew
