Tom-Newton commented on issue #44308: URL: https://github.com/apache/arrow/issues/44308#issuecomment-2541779144
> The option 1 is better, right? Let's try the option 1. I thought so too but, after trying it, now I'm not so sure. I checked a few of the Azure SDKs (C++, Python, and golang) and it actually looks like none of them expose the [Copy Blob](https://learn.microsoft.com/en-us/rest/api/storageservices/copy-blob?tabs=microsoft-entra-id) API. I found the `SourceAuthorization` option which should allow configuring different authentication to the source when using [Copy Blob From URL](https://learn.microsoft.com/en-us/rest/api/storageservices/copy-blob-from-url?tabs=microsoft-entra-id). Plus some some [examples of how to use it in `azcopy`](https://github.com/Azure/azure-storage-azcopy/blob/ac9320275a2d5fcf590794c90489eee6a07458fd/ste/mgr-JobPartTransferMgr.go#L970-L987). There also seem to be a string of changes in `azcopy` to try to get this working correctly, because of problems like making too many token requests or using a cached token after it expired. The current implementation adds a custom http policy when creating the Azure client, which intercepts requests that include an empty `x-ms-copy-source-authorization` and populate it https://github.com/Azure/azure-storage-azcopy/blob/7f5ab059c28b09841d009d28b7510d4fcb167dc8/ste/sourceAuthPolicy.go#L47-L77. This is all looking rather complicated and I don't think we can generate the required bearer tokens with SAS or account key auth. So I think I'll stick with generating SAS tokens, and I think I will just bump their expiry a bit as a simple mitigation for the [problem mentioned in azcopy](https://github.com/Azure/azure-storage-azcopy/pull/2513) where tokens expired during retries. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: github-unsubscr...@arrow.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
