sunildataengineer commented on PR #72336: URL: https://github.com/apache/airflow/pull/72336#issuecomment-5493405704
Hi @dabla, thanks so much for picking this up and pushing it forward. I originally worked on the deferrable mode for SFTPOperator, and I know firsthand how complex it was to get right. This feature is really important to me and to the Airflow community. Here's why: Deferrable operators free up worker slots during long I/O operations, which is huge for production Airflow instances where resources are tight For users transferring large files over SFTP, this means they can run more tasks in parallel without spinning up more workers — better cost efficiency The refactoring you did to share transfer logic between sync and async paths is exactly the right approach — it eliminates bugs and makes maintenance easier going forward I faced a lot of challenges getting the original implementation right — edge cases around directory traversal, async error handling, concurrency limits. Your version looks solid and well-tested. I'd really appreciate if this could get merged soon. I know there are a lot of PRs in the queue, but this feature unlocks better resource utilization for anyone doing file transfers in their Airflow DAGs, which is a pretty common use case. Thanks for sticking with this and making it production-ready. The Airflow ecosystem needs more operators like this. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
