firasbouzazi opened a new pull request, #77271: URL: https://github.com/apache/airflow/pull/77271
`DatabricksSQLStatementsSensor` with the default `deferrable=False` submitted the statement and then finished successfully straight away: its `execute()` replaced `BaseSensorOperator.execute()` and only deferred when `deferrable=True`, so `poke()` never ran. Downstream tasks started while the statement was still running, or after it had failed, and `timeout` never applied. The sensor now hands over to `BaseSensorOperator.execute()` when it is not deferring, so it polls the statement like any other sensor. The deferrable path is unchanged. `mode="reschedule"` with `statement` is now refused with a `ValueError`: every reschedule runs `execute()` again in a new process, so the sensor would submit the statement again each time. Reschedule mode keeps working with `statement_id`. Until now this combination never polled at all, so no working Dag relies on it, but it is a behaviour change worth a look. Also fixes the failure message, which printed the whole statement state where the statement ID should be. Tests: the new tests fail without the change. `test_exec_success` used to say that the non-deferrable path behaves the same whether the statement succeeds or fails; it now mocks a finished statement, as do the four `query_tags` tests, since the sensor now polls. All 1406 Databricks provider unit tests and the pre-commit static checks pass locally. closes: #77268 --- ##### Was generative AI tooling used to co-author this PR? - [X] Yes — Claude Code Generated-by: Claude Code following [the guidelines](https://github.com/apache/airflow/blob/main/contributing-docs/05_pull_requests.rst#gen-ai-assisted-contributions) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
