Tom-Newton commented on issue #44308: URL: https://github.com/apache/arrow/issues/44308#issuecomment-2538945799
I have it working to some extent, but as I feared there is a complexity with `CopyFiles` because its implemented by creating a SAS token at runtime https://github.com/apache/arrow/pull/41276. When authenticated with a SAS token already we cannot generate a new SAS token in the current way (I tested that it does work with Azure CLI auth so the problem is not just broken code that we still need to test https://github.com/apache/arrow/issues/41315). There are 2 options: 1. Stop using SAS token inside `CopyFiles` 1. Normally a SAS token is only needed when doing a copy from one Azure storage account to another Azure storage account. Within a single storage account we should be able to use the [Copy Blob](https://learn.microsoft.com/en-us/rest/api/storageservices/copy-blob?tabs=microsoft-entra-id) API instead of [Copy Blob From URL](https://learn.microsoft.com/en-us/rest/api/storageservices/copy-blob-from-url?tabs=microsoft-entra-id) but I'm not sure if this is actually exposed by the C++ SDK. 1. I'm a bit concerned by the current implementation that generates a new SAS token for every blob. Azure tends to have tight rate limits on APIs to generate authentication tokens, so I suspect a call to `CopyFiles` on a large directory would have problems. 1. We can close https://github.com/apache/arrow/issues/41315 4. Use the SAS token we already have 1. Definitely possible. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: github-unsubscr...@arrow.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
