potiuk opened a new issue, #66466: URL: https://github.com/apache/airflow/issues/66466
## Background PR #66452 caps `azure-batch<15.0.0` in `providers/microsoft/azure/pyproject.toml` because the [azure-batch 15.x release line](https://github.com/Azure/azure-sdk-for-python/blob/main/sdk/batch/azure-batch/CHANGELOG.md) (15.1.0 GA on PyPI 2026-05-01) is a complete rewrite (Azure SDK "track 2") that removes `BatchServiceClient`, `batch_auth`, and the model classes that `AzureBatchHook` and `AzureBatchOperator` currently depend on. The cap is a **workaround**, not a fix. This issue tracks the work needed to lift it. ## What needs to change ### 1. `providers/microsoft/azure/src/airflow/providers/microsoft/azure/hooks/batch.py` The hook currently imports: ```python from azure.batch import BatchServiceClient, batch_auth, models as batch_models from azure.batch.models import JobAddParameter, PoolAddParameter, TaskAddParameter ``` Track-2 equivalents to migrate to: | Old (≤14.x) | New (15.x) | |---|---| | `BatchServiceClient(credentials, batch_url=...)` | `BatchClient(endpoint=..., credential=...)` (sync) — module path TBD per release | | `batch_auth.SharedKeyCredentials(login, key)` | `AzureNamedKeyCredential` (or whatever shared-key support track 2 ships) | | `batch_models.PoolAddParameter` | `BatchPoolCreateContent` (typical track-2 rename pattern) | | `batch_models.JobAddParameter` | `BatchJobCreateContent` | | `batch_models.TaskAddParameter` | `BatchTaskCreateContent` | | `batch_models.VirtualMachineConfiguration` | renamed in track 2 — verify in changelog | | `batch_models.CloudServiceConfiguration` | **removed** in 15.x (cloud-service VMs deprecated by Azure) — drop the code path | | `batch_models.ImageReference` | `BatchVmImageReference` | | `batch_models.BatchErrorException` | `azure.core.exceptions.HttpResponseError` (track-2 unified exceptions) | | `batch_models.AccountListSupportedImagesOptions` | replaced with kwargs on the list method | | `batch_models.ComputeNodeState` | enum renamed — check 15.x docs | | `batch_client.pool.list()` etc. | sync client methods may rename to `list_pools()` etc. (track-2 convention) | ### 2. `providers/microsoft/azure/src/airflow/providers/microsoft/azure/operators/batch.py` `AzureBatchOperator`'s public constructor exposes 8 model types as parameter type hints: ```python batch_job_manager_task: batch_models.JobManagerTask | None = None, batch_job_preparation_task: batch_models.JobPreparationTask | None = None, batch_job_release_task: batch_models.JobReleaseTask | None = None, batch_task_container_settings: batch_models.TaskContainerSettings | None = None, batch_start_task: batch_models.StartTask | None = None, batch_task_resource_files: list[batch_models.ResourceFile] | None = None, batch_task_output_files: list[batch_models.OutputFile] | None = None, batch_task_user_identity: batch_models.UserIdentity | None = None, ``` These are **part of the operator's public API** — renaming the types is a breaking change for any DAG that imports them. Migration needs: - A deprecation cycle: keep old names re-exported as aliases for one provider release with a `DeprecationWarning`, then remove. - Or a major-version bump of the `microsoft.azure` provider. ### 3. Tests - `providers/microsoft/azure/tests/unit/microsoft/azure/hooks/test_batch.py` - `providers/microsoft/azure/tests/unit/microsoft/azure/operators/test_batch.py` These mock the old SDK's class names. They'll need to be re-mocked against the new client surface. ### 4. Docs - `providers/microsoft/azure/docs/operators/batch.rst` (if it exists) — example snippets reference old model names. - `providers/microsoft/azure/docs/index.rst` — bump the `azure-batch` requirement back to a single lower bound once the cap is removed. ### 5. Authentication paths The current hook supports two auth modes: - Shared-key (`batch_auth.SharedKeyCredentials`) - Azure Identity (`AzureIdentityCredentialAdapter`) Both need re-validation against the 15.x client. The `AzureIdentityCredentialAdapter` shim in `airflow.providers.microsoft.azure.hooks._azure_credential` may be obsolete in track 2 (the new client accepts `azure-identity` credentials natively). ## Acceptance criteria The cap can be removed (`azure-batch>=15.0.0` allowed) when all of the following hold: - [ ] `AzureBatchHook` works against `azure-batch >= 15.0.0` with both shared-key and identity credentials. - [ ] `AzureBatchOperator` accepts the new model types. - [ ] `providers/microsoft/azure/tests/unit/microsoft/azure/{hooks,operators}/test_batch.py` pass against the new SDK. - [ ] Compat tests (`Compat 2.11.1:P3.10`, `Compat 3.0.6:P3.10`, `Compat 3.1.8:P3.10`, `Compat 3.2.1:P3.10`) pass — these run the provider's tests against past Airflow versions, and the migration must keep import-time compatibility with all of them. - [ ] The deprecation strategy for the renamed model-type hints in `AzureBatchOperator`'s public signature is decided and documented (one-release alias warnings vs. straight provider major bump). - [ ] Changelog entry in `providers/microsoft/azure/docs/changelog.rst` describing the upgrade and any breaking surface. - [ ] The `<15.0.0` cap is removed from `providers/microsoft/azure/pyproject.toml` and the comment pointing to this issue is removed. - [ ] The matching cap in `providers/microsoft/azure/docs/index.rst` is removed. ## References - PR #66452 — applied the cap. - [azure-batch CHANGELOG](https://github.com/Azure/azure-sdk-for-python/blob/main/sdk/batch/azure-batch/CHANGELOG.md) — upstream notes on the rewrite. - [Azure SDK track-2 design guidelines](https://azure.github.io/azure-sdk/python/guidelines/) — naming conventions for client + model renames. ## Notes on the cap location When this issue is opened, please update the workaround comment in `providers/microsoft/azure/pyproject.toml` (currently says "Lifting the upper bound cap needs a full hook rewrite") to also point at this issue's URL, per the project's convention for tracked workarounds. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
