tannnmayy commented on PR #71557: URL: https://github.com/apache/airflow/pull/71557#issuecomment-5551096449
Closing this in favour of #72336. @dabla was right that the hook needed restructuring first. Without an `async def transfer()` on `SFTPHookAsync`, the deferrable path was always going to end up duplicating transfer logic in the trigger and freeing the worker slot without actually improving throughput. #72336 does that properly, and building on @sunildataengineer's original work is the right foundation for it. Thanks @dabla for the unusually detailed review — the breakdown of the `create_intermediate_dirs` / `confirm` / `prefetch` divergence between the sync and deferred paths was the part I had wrong, and it was useful to have it spelled out. And thanks @sunildataengineer for the kind words earlier in this thread. I'll follow #72336 and am happy to help test it if that's useful. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
