firasbouzazi opened a new pull request, #77271:
URL: https://github.com/apache/airflow/pull/77271

   `DatabricksSQLStatementsSensor` with the default `deferrable=False` 
submitted the statement and then finished successfully straight away: its 
`execute()` replaced `BaseSensorOperator.execute()` and only deferred when 
`deferrable=True`, so `poke()` never ran. Downstream tasks started while the 
statement was still running, or after it had failed, and `timeout` never 
applied.
   
   The sensor now hands over to `BaseSensorOperator.execute()` when it is not 
deferring, so it polls the statement like any other sensor. The deferrable path 
is unchanged.
   
   `mode="reschedule"` with `statement` is now refused with a `ValueError`: 
every reschedule runs `execute()` again in a new process, so the sensor would 
submit the statement again each time. Reschedule mode keeps working with 
`statement_id`. Until now this combination never polled at all, so no working 
Dag relies on it, but it is a behaviour change worth a look.
   
   Also fixes the failure message, which printed the whole statement state 
where the statement ID should be.
   
   Tests: the new tests fail without the change. `test_exec_success` used to 
say that the non-deferrable path behaves the same whether the statement 
succeeds or fails; it now mocks a finished statement, as do the four 
`query_tags` tests, since the sensor now polls. All 1406 Databricks provider 
unit tests and the pre-commit static checks pass locally.
   
   closes: #77268
   
   ---
   
   ##### Was generative AI tooling used to co-author this PR?
   
   - [X] Yes — Claude Code
   
   Generated-by: Claude Code following [the 
guidelines](https://github.com/apache/airflow/blob/main/contributing-docs/05_pull_requests.rst#gen-ai-assisted-contributions)
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]

Reply via email to