chinwobble commented on issue #18999: URL: https://github.com/apache/airflow/issues/18999#issuecomment-972378424
@eskarimov Happy to review your code. I'm not the best python dev, so I think I could learn a thing or two from you. https://github.com/apache/airflow/issues/18999#issuecomment-958449899 These deferrable operators are a good first step in making airflow more scalable. However, there are a few issues that make me feel its not ready yet: - task duration during deferral is not counted - for the databricks use case, there is an unnecessary number of processes being started / killed - More polling is introduced, for our pipeline that takes 2.5 hours / day, an extra 20 mins was added after implementing deferrable operators. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
