flo076 commented on PR #73454: URL: https://github.com/apache/airflow/pull/73454#issuecomment-6456528778
Thanks for clarifying. The impact for us is broader than BigQuery. We have observed unexpected interruptions affecting multiple deferrable GCP operators, including BigQuery and Dataproc. In the BigQuery case, audit logs confirm explicit CancelJob calls while the task was deferred; two successive attempts of the same task ended in cancellation, so a retry did not reliably recover it. We cannot yet prove that trigger reassignment caused every incident. Nevertheless, the core failure mode fixed by this PR affects multiple remote-job triggers, and makes deferrable mode effectively unsafe for our affected workloads on Airflow 3.3.1. Falling back to non-deferrable execution consumes workers for the duration of these jobs. Given this operational impact, could a 3.3.x backport be reconsidered? If that is not possible, is there a supported mitigation for managed environments that cannot move to 3.4.0 immediately? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
