@ash since we’re dropping 3.6 support soon anyways, would we even need this warning? Based on the POC it seems that this involves core changes, so I imagine if this is only 2.2.X compatible then ending py3.6 support for 2.2.X would be sufficient? (unless truly ending 3.6 support would need to wait for a 3.X release?)
On Mon, Apr 26, 2021 at 9:08 AM, Andrew Godwin <andrew.god...@astronomer.io.invalid> wrote: Yes - the prototype currently has a `airflow.utils.asyncio` module that tries to adapt things for 3.6, which I will likely just replace with a series of very polite errors about needing 3.7+. Andrew On Mon, Apr 26, 2021 at 9:50 AM Ash Berlin-Taylor < a...@apache.org [a...@apache.org] > wrote: Cool! I see that you are using async code for it in the draft PR, and I remember you saying somewhere that Python 3.6 (which is still supported) doesn't have great support for this. Should we have an explicit check for Py 3.6 and refuse to allow triggers to be run there? (and not let `trigger` process run either. -ash On Mon, Apr 26 2021 at 14:37:23 +0100, Kaxil Naik < kaxiln...@gmail.com [kaxiln...@gmail.com] > wrote: Thanks Andrew, that answers my questions. If we don't have any other questions by the end of the week we should start a VOTE. Regards, Kaxil On Tue, Apr 20, 2021 at 2:24 AM Andrew Godwin <andrew.god...@astronomer.io.invalid> wrote: Thanks Kaxil - notes on those two things: - A timeout is probably a reasonable thing to have in most situations, as it gives you that little bit of self-healing ability (I put a created timestamp into the Trigger schema partially out of this kind of caution). I'll update the AIP to mention that triggers will come with an optional timeout. - Correct, users who don't want the new process shouldn't be affected at all. The main bad failure case here is that if you don't have the process and then try to use a deferred operator, it would silently hang forever; my plan was to take a cue from the code in the webserver that warns you the scheduler isn't running, and do something similar for triggers, somewhere (it can't be an immediate error when you try to run the task, as the triggerer process may just be down for a few seconds for a redeploy, or similar). Andrew On Mon, Apr 19, 2021 at 6:10 PM Kaxil Naik < kaxiln...@gmail.com [kaxiln...@gmail.com] > wrote: +1 This is awesome Andrew, this is going to be a huge cost saver. The AIP is quite detailed and the Draft PR certainly helps. Just minor comments: 1. Should we have a timeout on publishing a Trigger / Task to the triggerer similar to https://airflow.apache.org/docs/apache-airflow/stable/configurations-ref.html#operation-timeout [https://airflow.apache.org/docs/apache-airflow/stable/configurations-ref.html#operation-timeout] 2. In How are users affected by the change? [https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=177050929#AIP40:Deferrable(%22Async%22)Operators-Howareusersaffectedbythechange?(e.g.DBupgraderequired?)] section, do the users who don't want to use the new process affected by it. Just confirming that if a user does not opt-in they are unaffected. [https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=177050929#AIP40:Deferrable(%22Async%22)Operators-Howareusersaffectedbythechange?(e.g.DBupgraderequired?)] Regards, Kaxil On Thu, Apr 15, 2021 at 9:34 PM Andrew Godwin <andrew.god...@astronomer.io.invalid> wrote: Hi all, After a period of designing and prototyping, I've completed a first draft of AIP-40 that I'd love to get some feedback on from the community: https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=177050929 [https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=177050929] This AIP proposes a way of adding what I'm calling "deferrable" Operators into Airflow - essentially, taking the goal of Smart Sensors and making it much more generalisable, where any Sensor or Operator can choose to "defer" its execution based on an asynchronous trigger, and where all the triggers run in one (or more) processes for efficiency. It also means that any Operator can be made deferrable in a backwards-compatible way should we wish to in future, though I'm not proposing that for a first release. It comes with a working prototype, too, should you wish to see what kind of code footprint this would have: https://github.com/apache/airflow/pull/15389 [https://github.com/apache/airflow/pull/15389] I personally think this would be a huge improvement for Airflow's efficiency - I suspect we might be able to reduce the amount of resources some Airflow installs use by over 50% if all their idling operators were ported to be deferrable - but I would welcome further opinions. Thanks, Andrew
