Not that soon - December 2021 is the EOL for Python 3.6, so I while it could be only 3.7+ feature, but warnings would still be nice. I don't think we discussed either timeline or scope for 2.1 or 2.2 but I think having them sooner both before December might be a good idea :)
J. On Mon, Apr 26, 2021 at 7:09 PM Daniel Imberman <daniel.imber...@gmail.com> wrote: > @ash since we’re dropping 3.6 support soon anyways, would we even need > this warning? Based on the POC it seems that this involves core changes, so > I imagine if this is only 2.2.X compatible then ending py3.6 support for > 2.2.X would be sufficient? (unless truly ending 3.6 support would need to > wait for a 3.X release?) > > On Mon, Apr 26, 2021 at 9:08 AM, Andrew Godwin > <andrew.god...@astronomer.io.invalid> wrote: > > Yes - the prototype currently has a `airflow.utils.asyncio` module that > tries to adapt things for 3.6, which I will likely just replace with a > series of very polite errors about needing 3.7+. > > Andrew > > On Mon, Apr 26, 2021 at 9:50 AM Ash Berlin-Taylor <a...@apache.org> wrote: > >> Cool! >> >> I see that you are using async code for it in the draft PR, and I >> remember you saying somewhere that Python 3.6 (which is still supported) >> doesn't have great support for this. >> >> Should we have an explicit check for Py 3.6 and refuse to allow triggers >> to be run there? (and not let `trigger` process run either. >> >> -ash >> >> On Mon, Apr 26 2021 at 14:37:23 +0100, Kaxil Naik <kaxiln...@gmail.com> >> wrote: >> >> Thanks Andrew, that answers my questions. >> >> If we don't have any other questions by the end of the week we should >> start a VOTE. >> >> Regards, >> Kaxil >> >> On Tue, Apr 20, 2021 at 2:24 AM Andrew Godwin >> <andrew.god...@astronomer.io.invalid> wrote: >> >>> Thanks Kaxil - notes on those two things: >>> >>> - A timeout is probably a reasonable thing to have in most situations, >>> as it gives you that little bit of self-healing ability (I put a created >>> timestamp into the Trigger schema partially out of this kind of caution). >>> I'll update the AIP to mention that triggers will come with an optional >>> timeout. >>> >>> - Correct, users who don't want the new process shouldn't be affected at >>> all. The main bad failure case here is that if you don't have the process >>> and then try to use a deferred operator, it would silently hang forever; my >>> plan was to take a cue from the code in the webserver that warns you the >>> scheduler isn't running, and do something similar for triggers, somewhere >>> (it can't be an immediate error when you try to run the task, as the >>> triggerer process may just be down for a few seconds for a redeploy, or >>> similar). >>> >>> Andrew >>> >>> On Mon, Apr 19, 2021 at 6:10 PM Kaxil Naik <kaxiln...@gmail.com> wrote: >>> >>>> +1 >>>> >>>> This is awesome Andrew, this is going to be a huge cost saver. >>>> >>>> The AIP is quite detailed and the Draft PR certainly helps. >>>> >>>> Just minor comments: >>>> >>>> >>>> 1. Should we have a timeout on publishing a Trigger / Task to the >>>> triggerer similar to >>>> >>>> https://airflow.apache.org/docs/apache-airflow/stable/configurations-ref.html#operation-timeout >>>> 2. In How are users affected by the change? >>>> >>>> <https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=177050929#AIP40:Deferrable(%22Async%22)Operators-Howareusersaffectedbythechange?(e.g.DBupgraderequired?)> >>>> section, do the users who don't want to use the new process affected by >>>> it. >>>> Just confirming that if a user does not opt-in they are unaffected. >>>> >>>> <https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=177050929#AIP40:Deferrable(%22Async%22)Operators-Howareusersaffectedbythechange?(e.g.DBupgraderequired?)> >>>> >>>> Regards, >>>> Kaxil >>>> >>>> On Thu, Apr 15, 2021 at 9:34 PM Andrew Godwin >>>> <andrew.god...@astronomer.io.invalid> wrote: >>>> >>>>> Hi all, >>>>> >>>>> After a period of designing and prototyping, I've completed a first >>>>> draft of AIP-40 that I'd love to get some feedback on from the community: >>>>> https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=177050929 >>>>> >>>>> This AIP proposes a way of adding what I'm calling "deferrable" >>>>> Operators into Airflow - essentially, taking the goal of Smart Sensors and >>>>> making it much more generalisable, where any Sensor or Operator can choose >>>>> to "defer" its execution based on an asynchronous trigger, and where all >>>>> the triggers run in one (or more) processes for efficiency. >>>>> >>>>> It also means that any Operator can be made deferrable in a >>>>> backwards-compatible way should we wish to in future, though I'm not >>>>> proposing that for a first release. >>>>> >>>>> It comes with a working prototype, too, should you wish to see what >>>>> kind of code footprint this would have: >>>>> https://github.com/apache/airflow/pull/15389 >>>>> >>>>> I personally think this would be a huge improvement for Airflow's >>>>> efficiency - I suspect we might be able to reduce the amount of resources >>>>> some Airflow installs use by over 50% if all their idling operators were >>>>> ported to be deferrable - but I would welcome further opinions. >>>>> >>>>> Thanks, >>>>> Andrew >>>>> >>>> -- +48 660 796 129
