Love it ! Thanks Jed :).
On Thu, Dec 9, 2021 at 9:35 PM Jed Cunningham
<[email protected]> wrote:
>
> Thanks everyone for the feedback. Being more user-centric is
certainly not a bad thing. (@Jarek, I didn't take your comments
personally.)
>
> I think I've addressed everything discussed here and in the PR,
so please take another look. High level, I've added a docs section
on migrating and linked to it across the board.
>
>
> On Thu, Dec 9, 2021 at 10:08 AM Kaxil Naik <[email protected]
<mailto:[email protected]>> wrote:
>>
>> In 100% agreement with the sentiments, Jarek and you have been
doing great work at improving that recently.
>>
>> Looking forward to a single Updating guide from Jed too.
>>
>>
>>
>> On Thu, Dec 9, 2021 at 5:04 PM Jarek Potiuk <[email protected]
<mailto:[email protected]>> wrote:
>>>
>>> This is cool, Jed. Looking forward to it!
>>>
>>> One more comment. Apologies if my words were understood as
"critique"
>>> of anyone's job. Jed - if it came to you as such - it was never
>>> intended. So - sincere apologies.
>>>
>>> I am as guilty as anyone of the "poor communication with our
users".
>>> So the "we have done quite a bad job in the past" applies to me
in my
>>> mind mostly (And I am personally embarrassed with some of
those). I've
>>> done a very poor job on that multiple times.
>>>
>>> My point is really - if there are - really small - things that
we can
>>> do now to improve what we have without introducing new ways
(which
>>> will take time I am sure) - why not?
>>> Creating a small chapter on "Why no instructions?" and "What
should I
>>> do now?" - thinking from our poor users' perspective - who have
no
>>> time to read the docs - takes almost no time. Linking to it
from the
>>> error message - also simple.
>>> This might prevent some questions from our users. So why not do
it?
>>>
>>> J.
>>>
>>> On Thu, Dec 9, 2021 at 5:56 PM Jed Cunningham
<[email protected]> wrote:
>>> >
>>> > As for UPDATING only being on github, I have a separate
proposal in that area coming soon. It likely won't be an issue come
time to release 2.3.0 👍.
>>> >
>>> > On Thu, Dec 9, 2021 at 9:34 AM Jarek Potiuk <[email protected]
<mailto:[email protected]>> wrote:
>>> >>
>>> >> And I agree with you :) (but with a twist).
>>> >>
>>> >> I do not say we should remove "UPDATING.md" information.
Not at all.
>>> >>
>>> >> Providing that:
>>> >>
>>> >> * UPDATING.md contains both
>>> >> * It is available in our User-facing docs
>>> >> * it has an anchor to this particular "piece of upgrading"
>>> >> * the deprecated error message has a direct link to it to
help to find it
>>> >>
>>> >> I (and our users I hope) would be perfectly happy.
>>> >>
>>> >> As far as I know. UPDATING.md is only in Github (I just
checked and I
>>> >> could not find in in airflow.apache.org
<http://airflow.apache.org/>. So by definition it's not a
>>> >> User documentation. It's developer documentation only.
>>> >>
>>> >> J
>>> >>
>>> >>
>>> >>
>>> >>
>>> >> On Thu, Dec 9, 2021 at 5:20 PM Kaxil Naik
<[email protected] <mailto:[email protected]>> wrote:
>>> >> >
>>> >> > Partially agree -- not completely.
>>> >> >
>>> >> > Firstly what I agree - (1) and (2) points from your email.
>>> >> >
>>> >> > Disagree the (3) point and the para after that.
>>> >> >
>>> >> > UPDATING.md is our source of breaking changes. Instead of
users just having to rely and checking "deprecation" for 100s of
commands, we should be helpful to users by also having a single page
where we list all the deprecations.
>>> >> >
>>> >> > That is another way of being helpful in finding the
"right" information and context quickly too. And "Guiding the users"
in a different way.
>>> >> >
>>> >> > On Thu, Dec 9, 2021 at 4:13 PM Jarek Potiuk
<[email protected] <mailto:[email protected]>> wrote:
>>> >> >>
>>> >> >> I really think about a chapter (Which was missing):
>>> >> >>
>>> >> >> "How should I approach this migration?"
>>> >> >>
>>> >> >> 1) explain why there is no 1-1 migration instruction
>>> >> >> 2) explain that for every smart sensor they need to use
or write
>>> >> >> deferrable operator
>>> >> >> 3) link to this information from "deprecation message"
they will see
>>> >> >> in the logs when they use smart sensor (rather than
relying on the
>>> >> >> fact that they will look at UPDATING.md and find the
right part
>>> >> >>
>>> >> >> That's it, Guiding the users. Being helpful in finding
the right
>>> >> >> information and context quickly (at the place where they
hit the error
>>> >> >> and not in one of the 100 pages of documentation that
they will only
>>> >> >> find by googling.
>>> >> >>
>>> >> >> J.
>>> >> >>
>>> >> >> On Thu, Dec 9, 2021 at 5:09 PM Kaxil Naik
<[email protected] <mailto:[email protected]>> wrote:
>>> >> >> >
>>> >> >> > Just as an FYI - the commit 18 hours ago on that PR
already had added "deprecation" in the docs too.
>>> >> >> >
>>> >> >> > Not only docs, but UPDATING.md, even in the Scheduler
logs, so kudos to Jed for taking care of it.
>>> >> >> >
>>> >> >> > So I don't agree with your comment or suggestion Jarek
at least in the context of this discussion as it makes me (at least)
read that the PR does not do those things.
>>> >> >> >
>>> >> >> > re: Tomek's question - it is a very valid question.
Unfortunately, I don't see a like-by-like replacement for DAG
Authors as different work needs to be done to write an Async
operator and make a sensor "smart sensor compatible".
>>> >> >> > However, agree that we try to be as clear as possible
on what a user might need to do - I just don't know what that would
be other than what I suggested in last email and would love the
feedback on the PR of what else can be included.
>>> >> >> >
>>> >> >> > Thanks.
>>> >> >> >
>>> >> >> > Regards,
>>> >> >> > Kaxil
>>> >> >> >
>>> >> >> > On Thu, Dec 9, 2021 at 4:00 PM Kaxil Naik
<[email protected] <mailto:[email protected]>> wrote:
>>> >> >> >>
>>> >> >> >> I don't think there is a 1-1 migration path. Async
operators supersede what Smart sensors were written to achieve -
Cost Savings.
>>> >> >> >>
>>> >> >> >> Smart Sensors were marked experimental feature for the
same reason and there are currently just two Sensors that are Smart
>>> >> >> >> sensors compatible.
>>> >> >> >>
>>> >> >> >> The only thing I can currently think of is writing an
async version of the Smart Sensor Hook and Operator differs based on
the underlying library that is used and
<https://airflow.apache.org/docs/apache-airflow/stable/concepts/deferring.html>
explains how you can write one. Also -
<https://airflow.apache.org/docs/apache-airflow/stable/concepts/deferring.html#smart-sensors>
>>> >> >> >>
>>> >> >> >>
>>> >> >> >>> I believe we have done quite a bad job in the past
assuming that our
>>> >> >> >>> users read all the discussions and AIPs we write.
They don't. They
>>> >> >> >>> need some guidance.
>>> >> >> >>
>>> >> >> >>
>>> >> >> >> Which instances? I am just curious to know what are
those bad instances where we "assumed" that our users read mailing
list and not covered it in UPDATING.md or docs.
>>> >> >> >>
>>> >> >> >> Regards,
>>> >> >> >> Kaxil
>>> >> >> >>
>>> >> >> >> On Thu, Dec 9, 2021 at 3:46 PM Jarek Potiuk
<[email protected] <mailto:[email protected]>> wrote:
>>> >> >> >>>
>>> >> >> >>> Extremely good point Tomek.
>>> >> >> >>>
>>> >> >> >>> Also as Ephraim pointed out in the PR - IMHO any time
when we do
>>> >> >> >>> deprecation we should have a note in our docs,
explaining at the very
>>> >> >> >>> least how the users should approach the migration as
correctly pointed
>>> >> >> >>> out by @turbaszek in the devlist.
>>> >> >> >>> I think this should be a standard of any deprecation
we do.
>>> >> >> >>>
>>> >> >> >>> I believe we have done quite a bad job in the past
assuming that our
>>> >> >> >>> users read all the discussions and AIPs we write.
They don't. They
>>> >> >> >>> need some guidance.
>>> >> >> >>>
>>> >> >> >>> J.
>>> >> >> >>>
>>> >> >> >>> On Wed, Dec 8, 2021 at 11:44 PM Tomasz Urbaszek
<[email protected] <mailto:[email protected]>> wrote:
>>> >> >> >>> >
>>> >> >> >>> > Do we have documentation about how to migrate from
smart sensors to deferrable operators?
