potiuk commented on issue #21218:
URL: https://github.com/apache/airflow/issues/21218#issuecomment-1030838373
This is not possible, because Hooks when instantiated might be created with
different init parameters.
Also I think the 'GenerlcTransfer` is not that useful and I think it adds
unnecessary complexity and generalization where we already have everything
needed to support the "generic transfer" by writing decorated tasks and using
hooks.
I think this is much nicer and more "new Airflow-y" way of implementing
transfer tasks using precisely the combination of Hooks and connections you
need. It's much more explicit:
```
@task
def transfer_task():
s3_hook = S3Hook(....params here ...)
gcs_hook = GCSHook(... params here ...)
// use .upload()/download() or streaming capabilities of the hooks to
perform the transfer
```
I will convert it into discussion and would love to hear what others think,
but I believe we should simply describ this as a valid pattern to do transfer
rather than investing into `GenericTransfer`.
WDYT @BasPH @dstandish @ashb @kaxil @mik-laj @turbaszek @olchas (I think
at some point of time you were involved in similar discussions).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
