1fanwang opened a new pull request, #72415: URL: https://github.com/apache/airflow/pull/72415
## Why If a worker exits after starting a synchronous Data Fusion pipeline, the retry starts another run while the first one keeps running. The retry now reconnects with the stored `runId`. Completed runs return without another submission. Failed, killed, or rejected runs start fresh. ## What The synchronous polling path uses AIP-103's `ResumableJobMixin`. It stores the exact `runId` before polling and restores that run with `get_pipeline_workflow`. Lookup errors stop the task rather than risk a duplicate. Asynchronous and deferrable execution keep their existing submission behavior. Airflow versions before 3.3 also keep the non-durable path. ## Tests The Data Fusion operator suite passed all 40 tests. Provider mypy and regular prek checks passed. <details> <summary>Task-state red/green proof</summary> ```console $ AIRFLOW_HOME=/tmp/airflow-datafusion-e2e-base .venv/bin/uv run --project providers/google python dev/datafusion_resumability_e2e.py --git-ref upstream/main --expect-recovery retry_result=run-002 completed_result=run-003 submissions=3 polls=3 store_events= recovery=FAIL $ AIRFLOW_HOME=/tmp/airflow-datafusion-e2e-current .venv/bin/uv run --project providers/google python dev/datafusion_resumability_e2e.py --expect-recovery retry_result=run-001 completed_result=run-001 submissions=1 polls=2 store_events=GET datafusion_pipeline_run_id=None | SET datafusion_pipeline_run_id=run-001 | GET datafusion_pipeline_run_id=run-001 | GET datafusion_pipeline_run_id=run-001 recovery=PASS red_status=1 green_status=0 ``` </details> ## Risk Only synchronous polling uses recovery. Unknown nonterminal states reconnect, and exact-run lookup failures propagate. --- ##### Was generative AI tooling used to co-author this PR? - [X] Yes, GitHub Copilot CLI (GPT-5.6 Sol) Generated-by: GitHub Copilot CLI (GPT-5.6 Sol) following [the guidelines](https://github.com/apache/airflow/blob/main/contributing-docs/05_pull_requests.rst#gen-ai-assisted-contributions) --- * Read the **[Pull Request Guidelines](https://github.com/apache/airflow/blob/main/contributing-docs/05_pull_requests.rst#pull-request-guidelines)** for more information. Note: commit author/co-author name and email in commits become permanently public when merged. * For fundamental code changes, an Airflow Improvement Proposal ([AIP](https://cwiki.apache.org/confluence/display/AIRFLOW/Airflow+Improvement+Proposals)) is needed. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
