jaketf commented on issue #6210: [AIRFLOW-5567] BaseAsyncOperator URL: https://github.com/apache/airflow/pull/6210#issuecomment-542428297 @potiuk I'd love to work on this further as I have time. I'd really like input on next steps: 1. I don't know how to fix the failing docs build. 1. Do people think the proposed XCom for tracking external resource id makes sense? 1. Are there additional features we need to make this viable? (See point 6) 1. Is there a good candidate operator to rework to use this? (there was discussion about dataflow and dataproc above but either the hooks are not set up well or the operators are actively being reworked) 1. Is there more test coverage needed? I tested what I considered new and not already tested in tests for BaseSensorOperator 1. Do we have any process for "scale testing" a feature like this? My concern is while this might pass basic unit tests, If I have tons of dags using async operators will it make the scheduler fall over? Certainly this could be mitigated by using different pool / priorities but should we be building a lower priority weight into BaseAsyncOperator defaults? Should there be a mechanism that specifies a submit priority weight vs a poke priority weight? 1. I think this feature would be awesome but will be confusing to users sometimes. We should probably add a section to the docs on BaseAsyncOperator, when it makes sense to use it and what assumptions it makes about a hook. Where would this belong? Should docs only PRs be separate? 1. Validate my assumption that this would not need a retroactive AIP.
---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: [email protected] With regards, Apache Git Services
