egotist0 opened a new pull request, #75432: URL: https://github.com/apache/airflow/pull/75432
<!-- SPDX-License-Identifier: Apache-2.0 --> On Airflow 3.x, `DataprocCreateBatchOperator(deferrable=True)` can remain deferred indefinitely despite `execution_timeout` when its trigger stalls. Airflow 2 already bounds deferrals in Core; this adds an operator-level backstop for Airflow 3.x. Pass the remaining execution budget to `defer()`, preserving unlimited waits when unset. An exhausted budget times out at the next trigger-timeout check. Some BigQuery operators pass the full duration; anchoring to `ti.start_date` here also counts time spent submitting the batch. The timeout bounds waiting without cancelling the batch. Retries with generated IDs, including `batch_id_prefix`, can start overlapping batches, increasing cost, and may duplicate non-idempotent writes. A fixed `batch_id` or a `request_id` that stays identical across retries can prevent duplicate submission; neither guarantees business-level idempotency. Tests: the three timeout-budget cases fail on the original implementation; the no-timeout case preserves existing behavior. All 123 Dataproc operator tests pass. The complete Google provider unit suite passes on the final diff: 5,530 passed, 66 skipped (Breeze, Python 3.12). related: #69678 (closed as not planned; this is a provider-level mitigation for that Core gap). --- ##### Was generative AI tooling used to co-author this PR? - [x] Yes — Codex (GPT-6) Generated-by: Codex (GPT-6) following [the guidelines](https://github.com/apache/airflow/blob/main/contributing-docs/05_pull_requests.rst#gen-ai-assisted-contributions) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
