SameerMesiah97 opened a new pull request, #61472: URL: https://github.com/apache/airflow/pull/61472
**Description** This change implements `execution_timeout` **semantics** for `DbtCloudRunJobOperator` when running in deferrable mode. Previously, when the operator deferred execution, `execution_timeout` was not enforced because the task process is no longer running and the scheduler cannot terminate it via SIGTERM. As a result, dbt Cloud jobs could continue running after the Airflow task exceeded its execution timeout. This update restores parity with non-deferrable execution by explicitly enforcing execution timeouts through the trigger/operator interaction. The operator now computes an absolute execution deadline derived from `execution_timeout` before deferring, the trigger emits a timeout event when that deadline is exceeded, and the operator cancels the dbt Cloud job and fails the task when the event is received. **Rationale** In non-deferrable mode, `execution_timeout` is enforced by the scheduler, which terminates the task process and invokes `on_kill()` to cancel the external dbt Cloud job. In deferrable mode, execution is handed off to a trigger running in the triggerer process, which does not have an associated worker process that can be terminated. However, **the absence of a worker process does not remove the requirement to honor task-level execution semantics**. From a user perspective, `execution_timeout` represents a hard task-level limit that should behave consistently regardless of whether execution is deferrable or not. Without explicit handling in the trigger/operator interaction, `execution_timeout` silently stops working in deferrable mode, leading to leaked dbt Cloud jobs and inconsistent behavior between deferrable and non-deferrable execution. Deferrable execution should adapt how timeouts are enforced, but not whether they are enforced. This change ensures `execution_timeout` semantics are preserved in deferrable mode and remain consistent with non-deferrable execution. **Notes** * The existing `timeout` parameter continues to limit only how long the operator waits for job completion and does not imply cancellation. * When both `execution_timeout` and `timeout` are set, the earlier deadline takes precedence. **Tests** * Added trigger-level tests asserting timeout events when the execution deadline is exceeded. * Added operator-level tests verifying job cancellation and task failure on execution timeout. * Extended trigger serialization tests to ensure the optional `execution_deadline` field is always serialized. **Documentation** * The docstring for `DbtCloudRunJobTrigger` has been updated to document the new `execution_deadline` parameter and clarify its behavior. * The docstring for `DbtCloudRunJobOperator` has been updated to clarify the behavior of the `timeout` parameter and distinguish it from task-level `execution_timeout`. **Backwards Compatibility** This change does not alter public APIs or method signatures. The runtime behavior changes in that dbt Cloud jobs are now explicitly cancelled when `execution_timeout` is reached during deferrable execution, whereas previously the job could continue running after the task timed out. Closes: #61467 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
