In 100% agreement with the sentiments, Jarek and you have been doing great work at improving that recently.
Looking forward to a single Updating guide from Jed too. On Thu, Dec 9, 2021 at 5:04 PM Jarek Potiuk <[email protected]> wrote: > This is cool, Jed. Looking forward to it! > > One more comment. Apologies if my words were understood as "critique" > of anyone's job. Jed - if it came to you as such - it was never > intended. So - sincere apologies. > > I am as guilty as anyone of the "poor communication with our users". > So the "we have done quite a bad job in the past" applies to me in my > mind mostly (And I am personally embarrassed with some of those). I've > done a very poor job on that multiple times. > > My point is really - if there are - really small - things that we can > do now to improve what we have without introducing new ways (which > will take time I am sure) - why not? > Creating a small chapter on "Why no instructions?" and "What should I > do now?" - thinking from our poor users' perspective - who have no > time to read the docs - takes almost no time. Linking to it from the > error message - also simple. > This might prevent some questions from our users. So why not do it? > > J. > > On Thu, Dec 9, 2021 at 5:56 PM Jed Cunningham <[email protected]> > wrote: > > > > As for UPDATING only being on github, I have a separate proposal in that > area coming soon. It likely won't be an issue come time to release 2.3.0 👍. > > > > On Thu, Dec 9, 2021 at 9:34 AM Jarek Potiuk <[email protected]> wrote: > >> > >> And I agree with you :) (but with a twist). > >> > >> I do not say we should remove "UPDATING.md" information. Not at all. > >> > >> Providing that: > >> > >> * UPDATING.md contains both > >> * It is available in our User-facing docs > >> * it has an anchor to this particular "piece of upgrading" > >> * the deprecated error message has a direct link to it to help to find > it > >> > >> I (and our users I hope) would be perfectly happy. > >> > >> As far as I know. UPDATING.md is only in Github (I just checked and I > >> could not find in in airflow.apache.org. So by definition it's not a > >> User documentation. It's developer documentation only. > >> > >> J > >> > >> > >> > >> > >> On Thu, Dec 9, 2021 at 5:20 PM Kaxil Naik <[email protected]> wrote: > >> > > >> > Partially agree -- not completely. > >> > > >> > Firstly what I agree - (1) and (2) points from your email. > >> > > >> > Disagree the (3) point and the para after that. > >> > > >> > UPDATING.md is our source of breaking changes. Instead of users just > having to rely and checking "deprecation" for 100s of commands, we should > be helpful to users by also having a single page where we list all the > deprecations. > >> > > >> > That is another way of being helpful in finding the "right" > information and context quickly too. And "Guiding the users" in a different > way. > >> > > >> > On Thu, Dec 9, 2021 at 4:13 PM Jarek Potiuk <[email protected]> wrote: > >> >> > >> >> I really think about a chapter (Which was missing): > >> >> > >> >> "How should I approach this migration?" > >> >> > >> >> 1) explain why there is no 1-1 migration instruction > >> >> 2) explain that for every smart sensor they need to use or write > >> >> deferrable operator > >> >> 3) link to this information from "deprecation message" they will see > >> >> in the logs when they use smart sensor (rather than relying on the > >> >> fact that they will look at UPDATING.md and find the right part > >> >> > >> >> That's it, Guiding the users. Being helpful in finding the right > >> >> information and context quickly (at the place where they hit the > error > >> >> and not in one of the 100 pages of documentation that they will only > >> >> find by googling. > >> >> > >> >> J. > >> >> > >> >> On Thu, Dec 9, 2021 at 5:09 PM Kaxil Naik <[email protected]> > wrote: > >> >> > > >> >> > Just as an FYI - the commit 18 hours ago on that PR already had > added "deprecation" in the docs too. > >> >> > > >> >> > Not only docs, but UPDATING.md, even in the Scheduler logs, so > kudos to Jed for taking care of it. > >> >> > > >> >> > So I don't agree with your comment or suggestion Jarek at least in > the context of this discussion as it makes me (at least) read that the PR > does not do those things. > >> >> > > >> >> > re: Tomek's question - it is a very valid question. Unfortunately, > I don't see a like-by-like replacement for DAG Authors as different work > needs to be done to write an Async operator and make a sensor "smart sensor > compatible". > >> >> > However, agree that we try to be as clear as possible on what a > user might need to do - I just don't know what that would be other than > what I suggested in last email and would love the feedback on the PR of > what else can be included. > >> >> > > >> >> > Thanks. > >> >> > > >> >> > Regards, > >> >> > Kaxil > >> >> > > >> >> > On Thu, Dec 9, 2021 at 4:00 PM Kaxil Naik <[email protected]> > wrote: > >> >> >> > >> >> >> I don't think there is a 1-1 migration path. Async operators > supersede what Smart sensors were written to achieve - Cost Savings. > >> >> >> > >> >> >> Smart Sensors were marked experimental feature for the same > reason and there are currently just two Sensors that are Smart > >> >> >> sensors compatible. > >> >> >> > >> >> >> The only thing I can currently think of is writing an async > version of the Smart Sensor Hook and Operator differs based on the > underlying library that is used and > https://airflow.apache.org/docs/apache-airflow/stable/concepts/deferring.html > explains how you can write one. Also - > https://airflow.apache.org/docs/apache-airflow/stable/concepts/deferring.html#smart-sensors > >> >> >> > >> >> >> > >> >> >>> I believe we have done quite a bad job in the past assuming that > our > >> >> >>> users read all the discussions and AIPs we write. They don't. > They > >> >> >>> need some guidance. > >> >> >> > >> >> >> > >> >> >> Which instances? I am just curious to know what are those bad > instances where we "assumed" that our users read mailing list and not > covered it in UPDATING.md or docs. > >> >> >> > >> >> >> Regards, > >> >> >> Kaxil > >> >> >> > >> >> >> On Thu, Dec 9, 2021 at 3:46 PM Jarek Potiuk <[email protected]> > wrote: > >> >> >>> > >> >> >>> Extremely good point Tomek. > >> >> >>> > >> >> >>> Also as Ephraim pointed out in the PR - IMHO any time when we do > >> >> >>> deprecation we should have a note in our docs, explaining at the > very > >> >> >>> least how the users should approach the migration as correctly > pointed > >> >> >>> out by @turbaszek in the devlist. > >> >> >>> I think this should be a standard of any deprecation we do. > >> >> >>> > >> >> >>> I believe we have done quite a bad job in the past assuming that > our > >> >> >>> users read all the discussions and AIPs we write. They don't. > They > >> >> >>> need some guidance. > >> >> >>> > >> >> >>> J. > >> >> >>> > >> >> >>> On Wed, Dec 8, 2021 at 11:44 PM Tomasz Urbaszek < > [email protected]> wrote: > >> >> >>> > > >> >> >>> > Do we have documentation about how to migrate from smart > sensors to deferrable operators? >
