venkata-pavani opened a new pull request, #68520:
URL: https://github.com/apache/airflow/pull/68520
## Summary
- Add `deferrable` parameter to `SFTPOperator`, allowing `get` and `put`
transfers of a single file (or list of single-file pairs, when `concurrency ==
1`) to be deferred to a trigger instead of blocking a worker slot while the
transfer is in progress.
- Add `SFTPOperatorTrigger` (in `airflow.providers.sftp.triggers.sftp`)
which performs the async file transfer via `SFTPHookAsync` and yields a
`TriggerEvent` on success or failure.
- `deferrable=True` raises `ValueError` at execute-time for unsupported
configurations:
- `operation == "delete"`
- `concurrency > 1`
- no `ssh_conn_id` available (neither explicit `ssh_conn_id` nor an
`sftp_hook` with one set)
This re-implements the design from #65480 (closed unmerged due to a rebase
issue), scoped down per discussion on #65475.
Closes #65475
## Test plan
- [x] New unit tests added in
`providers/sftp/tests/unit/sftp/triggers/test_sftp.py`
(`TestSFTPOperatorTrigger`) covering serialization, get/put success,
intermediate directory creation, and failure events.
- [x] New unit tests added in
`providers/sftp/tests/unit/sftp/operators/test_sftp.py` covering deferral
behavior, the `ValueError` guard rails, and `execute_complete`.
- [ ] CI test suite (could not run pytest locally on this Windows checkout
due to POSIX-only dependencies in airflow-core: `fcntl`, `os.register_at_fork`,
and symlinked `_shared` packages).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
