pankajastro opened a new pull request, #73374: URL: https://github.com/apache/airflow/pull/73374
DataFusion credential resolution only handled AWS (and, in a sibling PR, GCS), so a wasb-backed `DataSourceConfig` failed with `Unknown connection type wasb` at registration, even though DataFusion's storage layer supports Azure Blob Storage. Adds: - `StorageType.AZURE` and `az://` URI recognition in `DataSourceConfig` - `AzureObjectStorageProvider` using DataFusion's `MicrosoftAzure` object store binding - A `wasb` branch in `_get_credentials` resolving, in order: Azure AD service principal (`tenant_id` extra + `login`/`password` as `client_id`/`client_secret`), SAS token (`sas_token` extra, parsed into query pairs), then shared key (`password`, or `shared_access_key`/`account_key` extra) — falling back to ambient/environment-based auth when none apply - `microsoft.azure` extra in `pyproject.toml`, docs updated to mention Azure alongside S3 **On the service-principal fields**: DataFusion's `MicrosoftAzure` binding ([`crates/core/src/store.rs`](https://github.com/apache/datafusion-python/blob/main/crates/core/src/store.rs)) panics if `client_id`/`client_secret`/`tenant_id` are only partially set — verified directly: ``` thread '<unnamed>' panicked at src/store.rs:114:17: client_id, client_secret, tenat_id must be all set or all None ``` (PyO3 converts this to a `PanicException` rather than crashing the process, but it's still an ugly failure mode.) So this PR only forwards all three when the connection genuinely has all three (`tenant_id` extra plus both `login` and `password`); otherwise it falls through to the next auth mode rather than risking a partial combination. **Known limitation**: `connection_string` auth and `DefaultAzureCredential`/managed-identity resolution (both supported by Airflow's own `WasbHook`) aren't covered — DataFusion's binding has no equivalent parameter for either, only the explicit credential fields above. --- ##### Was generative AI tooling used to co-author this PR? - [X] Yes — Claude Code (Sonnet 5) Generated-by: Claude Code (Sonnet 5) following [the guidelines](https://github.com/apache/airflow/blob/main/contributing-docs/05_pull_requests.rst#gen-ai-assisted-contributions) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
