krishnakaanchan-png opened a new pull request, #3936: URL: https://github.com/apache/iceberg-python/pull/3936
Closes #2698 Builds on #3884, that one needs to go in first. Only the second commit here is new. # Rationale for this change This is the other half of #2698. #3884 stops the account leaking into the path. This one lets the account be read from the location at all. `_initialize_azure_fs` took no netloc, unlike `_initialize_s3_fs(netloc)` and `_initialize_hdfs_fs(scheme, netloc)` next to it. So `account_name` could only come from `adls.account-name`, and the account in an `abfss://<container>@<account>.dfs.core.windows.net/...` location was dropped. This leads to two things. A location without the property set failed with `ArrowInvalid: AzureOptions doesn't contain a valid account name`, and one FileIO could not serve two storage accounts. Now the netloc is passed through and the account comes from its host part when the property is not set. On precedence, an explicit `adls.account-name` still wins over the account in the location. I asked about this on #2698 and did not get a reply, so I have kept it the same as `FsspecFileIO`. That is the conservative choice, existing single account setups behave exactly as before. If you would rather it raised on a mismatch, tell me and I will change it. The fallback is gated on the userinfo part being present, not on the hostname. On the container only form `abfss://warehouse/f.parquet` the netloc is the container, so inferring an account from it would give `warehouse` and break the existing ADLS tests. Taking the account as the first label of the host holds for standard endpoints, private endpoints, sovereign clouds and the DNS zone endpoints. It does not hold for a custom domain on the blob endpoint. `FsspecFileIO` has the same limitation, so I have not tried to solve it here. ## Are these changes tested? Yes. I have added 4 new tests. The account coming from the location across all four ADLS schemes, an explicit property winning over the location, a container only netloc giving no account, and one FileIO serving two accounts and getting a separate filesystem for each. `make lint` is clean and `make test` passes. ## Are there any user-facing changes? Yes. `abfs[s]` and `wasb[s]` locations that carry the account now work under `PyArrowFileIO` without `adls.account-name` being set, where earlier they failed with `AzureOptions doesn't contain a valid account name`. A single FileIO can also serve more than one storage account now. Nothing changes when `adls.account-name` is set, or for the container only form. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
