dabla commented on code in PR #68298:
URL: https://github.com/apache/airflow/pull/68298#discussion_r3452562174
##########
providers/sftp/src/airflow/providers/sftp/hooks/sftp.py:
##########
@@ -735,6 +706,75 @@ def get_files_by_pattern(self, path, fnmatch_pattern) ->
list[str]:
return matched_files
+ def transfer(
Review Comment:
I don't like the fact that the transfer method of the SFTPHookAsync just
calls the sync methods on the sync SFTPHook but with the sync_to_async wrapper,
this to me also looks like an anti-pattern. This will surely work but doesn't
add any performance improvements. The transfer method of the SFTPHookAsync
should also call the native async methods of the hook. Unfortunately, the
SFTPHookAsync doesn't have the retrieve_directory_concurrently and
store_directory_concurrently methods, but that could be an opportunity to add
those in native async and using asyncio.gather, as those will be far more
performant than the sync counter parts which are using multi threading and
which aren't really appropriate for I/O related multiplexing.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
