uranusjr commented on PR #63500: URL: https://github.com/apache/airflow/pull/63500#issuecomment-4059913587
I feel we should also discuss when you should _not_ use one of the other. For example * You probaly shouldn’t prefer async over defer if the wait can be long (since an async operator is kept running in a process, which can waste resource). * You probably shouldn’t use defer if deferring happens many times and are generally short (even if you write a custom deferrable operator) since it stops the process every time it defers and restarts everything it resumes, which can introduce a lot of overhead. Also, the mention about existing deferrable operator can be a bit misleading IMO. If your use case is better for defer, and there’s no existing deferrable operator available, your best course would be to write one yourself, not to use async. It should depend more about the use case, not what Airflow has available. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
