kunaljubce commented on issue #42497: URL: https://github.com/apache/airflow/issues/42497#issuecomment-2376646005
The cross-storage account functionality has an added layer of complexity since, unlike in AWS, Azure implements a 3 tiered folder structure. AWS: Account (Auth) --> S3 Bucket --> Prefix --> Keys Azure: Account (Auth) --> Storage Account (Auth) --> Containers --> Blob Prefix --> Blob objects In Azure, due to the need to have authentication setup at a storage account level, it becomes imperative to create different instances of WasbHook for different storage accounts, which is not the case in AWS with the S3Hook. My plan is to handle this with an optional parameter, wherein users can mention the destination storage account details/hook for the dest storage account and the file copy would happen accordingly. Please let me know your thoughts! -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
