Vamsi-klu opened a new pull request, #70088: URL: https://github.com/apache/airflow/pull/70088
This adds first-class operators for starting and stopping existing Databricks SQL warehouses, including optional polling until the requested lifecycle state is reached. related: #21377 ### Problem Airflow's Databricks provider can execute SQL against a warehouse, but it has no first-class way to manage an existing warehouse's start/stop lifecycle. Dag authors currently need custom REST calls around their SQL tasks. ### What changed - Add `DatabricksHook` methods for retrieving, starting, and stopping a warehouse through the Databricks SQL Warehouses API. - Add a validated `WarehouseState` model for the six documented lifecycle states. - Add `DatabricksStartWarehouseOperator` and `DatabricksStopWarehouseOperator` with idempotent pre-checks, optional waiting, monotonic deadlines, and explicit terminal-state errors. - Register the operators in provider metadata and add a how-to guide plus a system-test example with unconditional stop cleanup. Warehouse IDs are embedded in the documented REST paths; no request-body workaround or new dependency is introduced. The implementation follows the [Databricks SQL Warehouses API](https://docs.databricks.com/api/workspace/warehouses). ### Scope This is the Phase 1 scope proposed on #21377: get/state/start/stop plus synchronous waiting. Create/delete, edit, warehouse-by-name resolution, async hooks, and deferrable operators remain outside this PR so the initial contribution stays reviewable and independently useful. ### Behavior and compatibility - Starting an already `RUNNING` warehouse and stopping an already `STOPPED` warehouse are no-ops. - Existing `STARTING`/`STOPPING` transitions are reused instead of issuing duplicate requests. - A start accepted while the warehouse still reports `STOPPING` continues polling; Databricks API transition rejections propagate unchanged. - Waiting uses `time.monotonic()` and never accepts a target state observed after the configured deadline. - The new operator validates templated warehouse IDs at execution time and remains import-compatible across supported Airflow versions through `common.compat.sdk`. ### Validation - `breeze run pytest providers/databricks/tests/unit/databricks/operators/test_databricks_warehouse.py -xvs` — 21 passed. - `breeze testing providers-tests --test-type "Providers[databricks]"` — 838 passed, 12 skipped. - Selective six-provider dependency matrix (`amazon, common.compat, common.sql, databricks, google, openlineage`) — 11,858 passed, 185 skipped. - `breeze run mypy <three changed Databricks source files>` — success, no issues. - Explicit nine-file `prek` pre-commit checks — passed. - Explicit nine-file `prek` manual checks — passed, including the providers mypy hook. - `breeze build-docs --docs-only --clean-build databricks` — documentation build successful; the generated guide contains both lifecycle examples. - `breeze run pytest providers/databricks/tests/system/databricks/example_databricks_sql_warehouse.py --collect-only -q` — 1 system test collected. - `breeze ci selective-check --commit-ref HEAD` — selected provider unit/compatibility tests, provider mypy, docs, Python scans, and the system-test path; no UI tests selected. ### Reviewer evidence This PR has no UI surface, so before/after screenshots and browser validation are not applicable. The REST boundary is covered with autospecced request assertions, operator behavior is covered with a specced hook, and the system example is import-validated. No Databricks workspace credentials were used or required for these deterministic lifecycle tests. The narrow Phase 1 scope was posted on the issue before implementation: https://github.com/apache/airflow/issues/21377#issuecomment-5014011201 --- ##### Was generative AI tooling used to co-author this PR? - [X] Yes — Codex (GPT-5) Generated-by: Codex (GPT-5) following [the guidelines](https://github.com/apache/airflow/blob/main/contributing-docs/05_pull_requests.rst#gen-ai-assisted-contributions) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
