I personally think the docs on deferrable are already comprehensive enough. I'll add more to UPDATING and the smart sensors docs to help guide users, however.
Also, keep in mind: - smart sensors are disabled by default - the only sensor Airflow ships with smart sensor support enabled by default is NamedHivePartitionSensor - even with the removal of smart sensors, DAGs will continue to run as-is, they just won't get the optimization of running the sensors in the "special" sensor runner On Thu, Dec 9, 2021 at 9:20 AM Kaxil Naik <[email protected]> wrote: > Partially agree -- not completely. > > Firstly what I agree - (1) and (2) points from your email. > > Disagree the (3) point and the para after that. > > UPDATING.md is our source of breaking changes. Instead of users just > having to rely and checking "deprecation" for 100s of commands, we should > be helpful to users by also having a single page where we list all the > deprecations. > > That is another way of being helpful in finding the "right" information > and context quickly too. And "Guiding the users" in a different way. > > On Thu, Dec 9, 2021 at 4:13 PM Jarek Potiuk <[email protected]> wrote: > >> I really think about a chapter (Which was missing): >> >> "How should I approach this migration?" >> >> 1) explain why there is no 1-1 migration instruction >> 2) explain that for every smart sensor they need to use or write >> deferrable operator >> 3) link to this information from "deprecation message" they will see >> in the logs when they use smart sensor (rather than relying on the >> fact that they will look at UPDATING.md and find the right part >> >> That's it, Guiding the users. Being helpful in finding the right >> information and context quickly (at the place where they hit the error >> and not in one of the 100 pages of documentation that they will only >> find by googling. >> >> J. >> >> On Thu, Dec 9, 2021 at 5:09 PM Kaxil Naik <[email protected]> wrote: >> > >> > Just as an FYI - the commit 18 hours ago on that PR already had added >> "deprecation" in the docs too. >> > >> > Not only docs, but UPDATING.md, even in the Scheduler logs, so kudos to >> Jed for taking care of it. >> > >> > So I don't agree with your comment or suggestion Jarek at least in the >> context of this discussion as it makes me (at least) read that the PR does >> not do those things. >> > >> > re: Tomek's question - it is a very valid question. Unfortunately, I >> don't see a like-by-like replacement for DAG Authors as different work >> needs to be done to write an Async operator and make a sensor "smart sensor >> compatible". >> > However, agree that we try to be as clear as possible on what a user >> might need to do - I just don't know what that would be other than what I >> suggested in last email and would love the feedback on the PR of what else >> can be included. >> > >> > Thanks. >> > >> > Regards, >> > Kaxil >> > >> > On Thu, Dec 9, 2021 at 4:00 PM Kaxil Naik <[email protected]> wrote: >> >> >> >> I don't think there is a 1-1 migration path. Async operators supersede >> what Smart sensors were written to achieve - Cost Savings. >> >> >> >> Smart Sensors were marked experimental feature for the same reason and >> there are currently just two Sensors that are Smart >> >> sensors compatible. >> >> >> >> The only thing I can currently think of is writing an async version of >> the Smart Sensor Hook and Operator differs based on the underlying library >> that is used and >> https://airflow.apache.org/docs/apache-airflow/stable/concepts/deferring.html >> explains how you can write one. Also - >> https://airflow.apache.org/docs/apache-airflow/stable/concepts/deferring.html#smart-sensors >> >> >> >> >> >>> I believe we have done quite a bad job in the past assuming that our >> >>> users read all the discussions and AIPs we write. They don't. They >> >>> need some guidance. >> >> >> >> >> >> Which instances? I am just curious to know what are those bad >> instances where we "assumed" that our users read mailing list and not >> covered it in UPDATING.md or docs. >> >> >> >> Regards, >> >> Kaxil >> >> >> >> On Thu, Dec 9, 2021 at 3:46 PM Jarek Potiuk <[email protected]> wrote: >> >>> >> >>> Extremely good point Tomek. >> >>> >> >>> Also as Ephraim pointed out in the PR - IMHO any time when we do >> >>> deprecation we should have a note in our docs, explaining at the very >> >>> least how the users should approach the migration as correctly pointed >> >>> out by @turbaszek in the devlist. >> >>> I think this should be a standard of any deprecation we do. >> >>> >> >>> I believe we have done quite a bad job in the past assuming that our >> >>> users read all the discussions and AIPs we write. They don't. They >> >>> need some guidance. >> >>> >> >>> J. >> >>> >> >>> On Wed, Dec 8, 2021 at 11:44 PM Tomasz Urbaszek <[email protected]> >> wrote: >> >>> > >> >>> > Do we have documentation about how to migrate from smart sensors to >> deferrable operators? >> >
