1fanwang opened a new pull request, #71522:
URL: https://github.com/apache/airflow/pull/71522
**TL;DR — operator extra links don't work per attempt** (#71471). After a
retry the button
under attempt 1 opens attempt 2's logs. It hits every bundled provider
linking to per-attempt
logs: EMR, Glue, Databricks, Dataproc, Livy.
Draft **C of three** for the same bug — pick a shape and I'll finish it and
close the others.
| | Approach | Trade |
|---|---|---|
| A | #71518 — a row per attempt in XCom | smallest; core owns a key
convention for every link |
| B | #71519 — the link opts in | no extra rows when unneeded; changes the
serialized DAG format |
| **C** | **this PR — the task state store** | **nothing about the clear
moves; 3.3+ only** |
## How
The worker writes each attempt's rendered link to the task state store as
well as to XCom. The
store is not cleared on retry — that is what it exists for, and what the
durable-execution work
in #69914 and #71211 relies on — so no exemption from the XCom clear is
needed. XCom keeps
holding the latest attempt for anything already reading it; the reader
prefers the store and
falls back to XCom.
`TaskScope` is keyed on `(dag_id, run_id, task_id, map_index)` with no
attempt, so the try is
still encoded in the key. The store buys immunity from the clear, not a
per-attempt dimension.
## Testing
Red to green on a real 3.3.0 install, store populated per attempt, XCom
holding the latest.
<details><summary>Raw output</summary>
Before, stock 3.3.0 — attempt 1 resolves to attempt 2's URL:
```
$ AIRFLOW__DATABASE__SQL_ALCHEMY_CONN=sqlite:///af.db python c33.py
try 1 -> https://logs/attempt-2
try 2 -> https://logs/attempt-2
```
After:
```
$ AIRFLOW__DATABASE__SQL_ALCHEMY_CONN=sqlite:///af.db python c33.py
try 1 -> https://logs/attempt-1
try 2 -> https://logs/attempt-2
```
</details>
## Two costs
It puts UI-render data in a store built for job identity, and the reader
queries
`task_state_store` directly rather than through the accessor workers use, so
a deployment
pointing `[state_store] backend` elsewhere would need the api-server to
reach it.
It also cannot ship on the 3.2 line: the store landed after 3.2.2, where
`airflow.models.task_state_store` and `airflow.sdk.state` are both absent. A
and B work there.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
