This is cool, Jed. Looking forward to it! One more comment. Apologies if my words were understood as "critique" of anyone's job. Jed - if it came to you as such - it was never intended. So - sincere apologies.
I am as guilty as anyone of the "poor communication with our users". So the "we have done quite a bad job in the past" applies to me in my mind mostly (And I am personally embarrassed with some of those). I've done a very poor job on that multiple times. My point is really - if there are - really small - things that we can do now to improve what we have without introducing new ways (which will take time I am sure) - why not? Creating a small chapter on "Why no instructions?" and "What should I do now?" - thinking from our poor users' perspective - who have no time to read the docs - takes almost no time. Linking to it from the error message - also simple. This might prevent some questions from our users. So why not do it? J. On Thu, Dec 9, 2021 at 5:56 PM Jed Cunningham <[email protected]> wrote: > > As for UPDATING only being on github, I have a separate proposal in that area > coming soon. It likely won't be an issue come time to release 2.3.0 👍. > > On Thu, Dec 9, 2021 at 9:34 AM Jarek Potiuk <[email protected]> wrote: >> >> And I agree with you :) (but with a twist). >> >> I do not say we should remove "UPDATING.md" information. Not at all. >> >> Providing that: >> >> * UPDATING.md contains both >> * It is available in our User-facing docs >> * it has an anchor to this particular "piece of upgrading" >> * the deprecated error message has a direct link to it to help to find it >> >> I (and our users I hope) would be perfectly happy. >> >> As far as I know. UPDATING.md is only in Github (I just checked and I >> could not find in in airflow.apache.org. So by definition it's not a >> User documentation. It's developer documentation only. >> >> J >> >> >> >> >> On Thu, Dec 9, 2021 at 5:20 PM Kaxil Naik <[email protected]> wrote: >> > >> > Partially agree -- not completely. >> > >> > Firstly what I agree - (1) and (2) points from your email. >> > >> > Disagree the (3) point and the para after that. >> > >> > UPDATING.md is our source of breaking changes. Instead of users just >> > having to rely and checking "deprecation" for 100s of commands, we should >> > be helpful to users by also having a single page where we list all the >> > deprecations. >> > >> > That is another way of being helpful in finding the "right" information >> > and context quickly too. And "Guiding the users" in a different way. >> > >> > On Thu, Dec 9, 2021 at 4:13 PM Jarek Potiuk <[email protected]> wrote: >> >> >> >> I really think about a chapter (Which was missing): >> >> >> >> "How should I approach this migration?" >> >> >> >> 1) explain why there is no 1-1 migration instruction >> >> 2) explain that for every smart sensor they need to use or write >> >> deferrable operator >> >> 3) link to this information from "deprecation message" they will see >> >> in the logs when they use smart sensor (rather than relying on the >> >> fact that they will look at UPDATING.md and find the right part >> >> >> >> That's it, Guiding the users. Being helpful in finding the right >> >> information and context quickly (at the place where they hit the error >> >> and not in one of the 100 pages of documentation that they will only >> >> find by googling. >> >> >> >> J. >> >> >> >> On Thu, Dec 9, 2021 at 5:09 PM Kaxil Naik <[email protected]> wrote: >> >> > >> >> > Just as an FYI - the commit 18 hours ago on that PR already had added >> >> > "deprecation" in the docs too. >> >> > >> >> > Not only docs, but UPDATING.md, even in the Scheduler logs, so kudos to >> >> > Jed for taking care of it. >> >> > >> >> > So I don't agree with your comment or suggestion Jarek at least in the >> >> > context of this discussion as it makes me (at least) read that the PR >> >> > does not do those things. >> >> > >> >> > re: Tomek's question - it is a very valid question. Unfortunately, I >> >> > don't see a like-by-like replacement for DAG Authors as different work >> >> > needs to be done to write an Async operator and make a sensor "smart >> >> > sensor compatible". >> >> > However, agree that we try to be as clear as possible on what a user >> >> > might need to do - I just don't know what that would be other than what >> >> > I suggested in last email and would love the feedback on the PR of what >> >> > else can be included. >> >> > >> >> > Thanks. >> >> > >> >> > Regards, >> >> > Kaxil >> >> > >> >> > On Thu, Dec 9, 2021 at 4:00 PM Kaxil Naik <[email protected]> wrote: >> >> >> >> >> >> I don't think there is a 1-1 migration path. Async operators supersede >> >> >> what Smart sensors were written to achieve - Cost Savings. >> >> >> >> >> >> Smart Sensors were marked experimental feature for the same reason and >> >> >> there are currently just two Sensors that are Smart >> >> >> sensors compatible. >> >> >> >> >> >> The only thing I can currently think of is writing an async version of >> >> >> the Smart Sensor Hook and Operator differs based on the underlying >> >> >> library that is used and >> >> >> https://airflow.apache.org/docs/apache-airflow/stable/concepts/deferring.html >> >> >> explains how you can write one. Also - >> >> >> https://airflow.apache.org/docs/apache-airflow/stable/concepts/deferring.html#smart-sensors >> >> >> >> >> >> >> >> >>> I believe we have done quite a bad job in the past assuming that our >> >> >>> users read all the discussions and AIPs we write. They don't. They >> >> >>> need some guidance. >> >> >> >> >> >> >> >> >> Which instances? I am just curious to know what are those bad >> >> >> instances where we "assumed" that our users read mailing list and not >> >> >> covered it in UPDATING.md or docs. >> >> >> >> >> >> Regards, >> >> >> Kaxil >> >> >> >> >> >> On Thu, Dec 9, 2021 at 3:46 PM Jarek Potiuk <[email protected]> wrote: >> >> >>> >> >> >>> Extremely good point Tomek. >> >> >>> >> >> >>> Also as Ephraim pointed out in the PR - IMHO any time when we do >> >> >>> deprecation we should have a note in our docs, explaining at the very >> >> >>> least how the users should approach the migration as correctly pointed >> >> >>> out by @turbaszek in the devlist. >> >> >>> I think this should be a standard of any deprecation we do. >> >> >>> >> >> >>> I believe we have done quite a bad job in the past assuming that our >> >> >>> users read all the discussions and AIPs we write. They don't. They >> >> >>> need some guidance. >> >> >>> >> >> >>> J. >> >> >>> >> >> >>> On Wed, Dec 8, 2021 at 11:44 PM Tomasz Urbaszek >> >> >>> <[email protected]> wrote: >> >> >>> > >> >> >>> > Do we have documentation about how to migrate from smart sensors to >> >> >>> > deferrable operators?
