AlexisBRENON commented on PR #33578: URL: https://github.com/apache/airflow/pull/33578#issuecomment-1689372781
> To be honest, I am not sure if we need it. I think there is a very little benefit from having such "common" provider. [...] > [...] it introduces coupling (i.e. all users of the common code become implicitly dependent). And it introduces all kinds of issues where change in the common code might lead to dubtle bugs in related providers. I understand your point of view and agree that having such generic class mixed with a lot of other ones make changes hard to do and to predict all the impacts. What about a less "common" provider, more focused on `fsspec.Filesystem` creation ? My actual use case is to move some data from one bucket to another. Currently, my operator take two URIs as arguments, as well as **one** connection ID (for GCS access in my case). But this requires some rework if I need: - to change one of the storage from GCS to S3 for exemple, - different credentials for each buckets. Having a `FileSystem` connection type would allow me to provision a connection for each of my storage service and using only two connections ID as arguments of the operator. > I yet fail to see what would be the benefits of introducing such common code. But maybe I am short-sited? Do you think that this use case would benefit from introducing a FileSystem connection, or do I mis-conceived my operator and/or workflow ? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
