sunildataengineer opened a new pull request, #65480:
URL: https://github.com/apache/airflow/pull/65480
What this PR does
Adds `deferrable=True` parameter to `SFTPOperator` which allows the
operator to defer execution to `SFTPOperatorTrigger`, freeing the
worker slot during file transfers instead of blocking it.
Currently, `SFTPOperator` blocks a worker slot for the entire duration
of every file transfer. For large file transfers, this wastes worker
resources unnecessarily.
Changes
- Added `SFTPOperatorTrigger` class in `triggers/sftp.py`
- Added `deferrable` parameter to `SFTPOperator.__init__()` with
default `False` (fully backward compatible)
- Added `self.defer()` call in `execute()` when `deferrable=True`
- Added `execute_complete()` callback method to handle trigger result
- Added 3 unit tests covering deferrable mode
Tests
- `test_sftp_operator_defers_when_deferrable_true`
- `test_sftp_operator_execute_complete_success`
- `test_sftp_operator_execute_complete_raises_on_error`
Closes #65475
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
