andrei-ionescu opened a new issue, #3283: URL: https://github.com/apache/arrow-rs/issues/3283
**Is your feature request related to a problem or challenge? Please describe what you are trying to do.** Azure Azure Data Lake Storage Gen2 is available since a good while. Here is [Microsoft Azure Data Lake Storage Gen2 REST APIs](https://learn.microsoft.com/en-us/rest/api/storageservices/data-lake-storage-gen2) documentation. I'm trying to access Azure ADLS Gen2 using `object_store` to read files. Then, as a next step that I want to try is to use DataFusion to read data from ADLS Gen2, process that data, and then write it back to the ADLS Gen2. Looking at these lines of code — [object_store/src/azure/mod.rs#L549-L574](https://github.com/apache/arrow-rs/blob/master/object_store/src/azure/mod.rs#L549-L574) — we can see that the ADLS Gen2 format bellow is not supported: ``` abfss://my_file_system@az_account_name.dfs.core.windows.net/my/path/id1/id2/ ``` **Describe the solution you'd like** Have support for Azure Data Lake Storage Gen2 in `object_store` library. **Describe alternatives you've considered** I did try to look into the [Azure SDK for Data Lake](https://github.com/Azure/azure-sdk-for-rust/tree/main/sdk/storage_datalake) but it does NOT seem to fit well with the `object_store` library that is used in DataFusion. My end goal is to read data parquet data from ADLS Gen2 with DataFusion. **Additional context** Here is an error that looks like it doesn't find the file and this is because it looks into the wrong place. ``` thread 'main' panicked at 'called `Result::unwrap()` on an `Err` value: Generic { store: "MicrosoftAzure", source: ListRequest { source: Error { retries: 0, message: "\u{feff}<?xml version=\"1.0\" encoding=\"utf-8\"?><Error><Code>AuthorizationPermissionMismatch</Code><Message>This request is not authorized to perform this operation using this permission.\nRequestId:013baf5b-501e-003a-3a90-095d29000000\nTime:2022-12-06T16:37:11.0059157Z</Message></Error>", source: Some(reqwest::Error { kind: Status(403), url: Url { scheme: "https", cannot_be_a_base: false, username: "", password: None, host: Some(Domain("az_account_name.blob.core.windows.net")), port: None, path: "/my_file_system", query: Some("restype=container&comp=list&prefix=my%2Fpath%2Fid1%2Fid2%2F"), fragment: None } }) } } }' ``` -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
