moomindani opened a new pull request, #72313: URL: https://github.com/apache/airflow/pull/72313
A Databricks run carries one entry per task *attempt*. `extract_failed_task_errors[_async]` walked every entry, so the failure list that ends up in a task's error message (`operators/databricks.py:159`, and the trigger event on the deferrable path) had two problems: * a task that Databricks retried was reported once per attempt, and `get_run_output` was called once per attempt as well * a task that failed on an earlier attempt but succeeded on the retry was still reported as failed Both go away by keeping only the attempt with the highest `attempt_number` per `task_key` before filtering for failures. The semantics I picked, in case anyone wants to argue for the other one: report the **last** attempt only. An earlier attempt's error does not describe the outcome of the task, and if the retry succeeded there is no failure to report at all. Deduplicating by `task_key` while keeping the first entry would fix the duplication but keep reporting recovered tasks as failures. Verified against a live workspace, with a job whose single task always fails and carries `max_retries: 1`: * the API returns two entries for that task — `attempt_number` 0 and 1, both `FAILED`, with different task run ids * after the change `extract_failed_task_errors` returns exactly one entry, carrying the last attempt's run id All three new unit tests fail without the change, and the provider's unit suite passes with it. related: #72304 --- ##### Was generative AI tooling used to co-author this PR? - [X] Yes — Claude Code (Opus 5) Generated-by: Claude Code (Opus 5) following [the guidelines](https://github.com/apache/airflow/blob/main/contributing-docs/05_pull_requests.rst#gen-ai-assisted-contributions) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
