abhishekmauryaKsolves opened a new pull request, #73647:
URL: https://github.com/apache/airflow/pull/73647
Closes #73585
## What
`SFTPHook.store_directory_concurrently()` and
`retrieve_directory_concurrently()` built each worker hook as
`SFTPHook(ssh_conn_id=self.ssh_conn_id)`, discarding every other constructor
override on the parent hook — `remote_host`, `port`, `username`, `password`,
`key_file`, proxy settings, timeouts, and `no_host_key_check`. Workers fell
back to whatever the underlying connection defined instead of the parent hook's
effective, already-resolved settings.
With an explicit `remote_host` set on the parent hook, for example, the
directory listing happened against that host but the worker transfers ran
against the connection's original host.
## How
Added `SFTPHook._build_worker_hook()`, a helper that constructs a new
`SFTPHook` from the parent hook's effective connection settings (constructor
overrides merged with the connection), and used it in both
`store_directory_concurrently()` and `retrieve_directory_concurrently()`
instead of re-constructing a bare `SFTPHook(ssh_conn_id=...)` per worker.
## Testing
Added two regression tests in `test_sftp.py`:
- `test_build_worker_hook_inherits_parent_overrides` — unit-tests
`_build_worker_hook()` directly, asserting it copies `remote_host`, `port`,
`username`, `password`, `key_file`, `conn_timeout`, `host_proxy_cmd`,
`no_host_key_check`, `allow_host_key_change`, and `look_for_keys`.
- `test_store_and_retrieve_directory_concurrently_use_parent_overrides` —
patches `_build_worker_hook` and asserts every worker hook built during a
concurrent store/retrieve transfer carries the parent's effective
`remote_host`/`port`/`username`.
Full `providers/sftp` unit test suite passes locally (127 passed).
---
<!-- SPDX-License-Identifier: Apache-2.0
https://www.apache.org/licenses/LICENSE-2.0 -->
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
