jaketf commented on issue #6210: [AIRFLOW-5567] BaseAsyncOperator
URL: https://github.com/apache/airflow/pull/6210#issuecomment-542428297
@potiuk I'd love to work on this further as I have time.
I'd really like input on next steps:
1. I don't know how to fix the failing docs build.
1. Do people think the proposed XCom for tracking external resource id makes
sense?
1. Are there additional features we need to make this viable? (See point 6)
1. Is there a good candidate operator to rework to use this? (there was
discussion about dataflow and dataproc above but either the hooks are not set
up well or the operators are actively being reworked)
1. Is there more test coverage needed? I tested what I considered new and
not already tested in tests for BaseSensorOperator
1. Do we have any process for "scale testing" a feature like this? My
concern is while this might pass basic unit tests, If I have tons of dags
using async operators will it make the scheduler fall over? Certainly this
could be mitigated by using different pool / priorities but should we be
building a lower priority weight into BaseAsyncOperator defaults? Should there
be a mechanism that specifies a submit priority weight vs a poke priority
weight?
1. I think this feature would be awesome but will be confusing to users
sometimes. We should probably add a section to the docs on BaseAsyncOperator,
when it makes sense to use it and what assumptions it makes about a hook. Where
would this belong? Should docs only PRs be separate?
1. Validate my assumption that this would not need a retroactive AIP.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
