shahar1 opened a new pull request, #77211:
URL: https://github.com/apache/airflow/pull/77211
Partition status is visible today only per consuming Dag (pending runs in
the Dags list) and as a "Partition Key" line on an asset event. Nothing answers
"which partitions of this asset are materialized, in progress or failed". A
failed or running producer run leaves no event, so it is invisible from the
asset page.
This adds a read-only **Partitions** tab to the asset page
(`/assets/:assetId/partitions`) and one UI-internal route, `GET
/ui/assets/{asset_id}/partitions`. There is no public REST API change, no
migration, and no Task SDK or scheduler change.
## What the tab shows
- A count per status, which also toggles a status filter.
- A day grid, only when every key on the page is a calendar day
(`YYYY-MM-DD`, with or without `T00:00:00`).
- A table: partition key, state, last producer run, last asset event, and
consumer runs (Dag runs that consumed the last event).
## Screenshots
From `breeze start-airflow`, with a daily partitioned producer and seeded
demo runs.
### Before
The asset page on `main`, same data. It has two tabs, and a partition
appears only as a "Partition Key" line on each event.
**>>> DRAG IMAGE HERE: 1-before-wide-light.png (Asset page before the
change, widened panel, light mode) <<<**
<img width="1440" height="900" alt="1-before-wide-light"
src="https://github.com/user-attachments/assets/a5caf1df-7fa2-46af-a509-558da83ffea0"
/>
**>>> DRAG IMAGE HERE: 2-before-wide-dark.png (Asset page before the change,
widened panel, dark mode) <<<**
<img width="1440" height="900" alt="2-before-wide-dark"
src="https://github.com/user-attachments/assets/0d9116fa-ddcd-4b03-acbd-85f52e74e15f"
/>
**>>> DRAG IMAGE HERE: 3-before-default-light.png (Asset page before the
change, default panel width, light mode) <<<**
<img width="1440" height="900" alt="3-before-default-light"
src="https://github.com/user-attachments/assets/ee0a51e6-eb6f-4a2e-a4ae-f5abac9f1b63"
/>
### After
Widened details panel, light and dark:
**>>> DRAG IMAGE HERE: 4-partitions-wide-light.png (Partitions tab, widened
panel, light mode) <<<**
<img width="1440" height="900" alt="4-partitions-wide-light"
src="https://github.com/user-attachments/assets/40a3c8de-113b-480c-831b-5ec6e87e59a6"
/>
**>>> DRAG IMAGE HERE: 5-partitions-wide-dark.png (Partitions tab, widened
panel, dark mode) <<<**
<img width="1440" height="900" alt="5-partitions-wide-dark"
src="https://github.com/user-attachments/assets/853a05a6-7ae3-4aa0-a3bc-76cde348375e"
/>
Default panel width (the table scrolls sideways):
**>>> DRAG IMAGE HERE: 6-partitions-default-light.png (Partitions tab,
default panel width, light mode) <<<**
<img width="1440" height="900" alt="6-partitions-default-light"
src="https://github.com/user-attachments/assets/4e7f0524-39be-42b1-bed2-4d504d9fa0ff"
/>
Status filter switched on for In Progress and Failed:
**>>> DRAG IMAGE HERE: 7-partitions-filtered-default-dark.png (Partitions
tab with the status filter on, dark mode) <<<**
<img width="1440" height="900" alt="7-partitions-filtered-default-dark"
src="https://github.com/user-attachments/assets/8642f64b-c39e-44ec-878b-2c68a8a4ef29"
/>
Keys that are not calendar days, so no day grid:
**>>> DRAG IMAGE HERE: 8-partitions-freeform-default-light-panel.png
(Partitions tab for an asset with free-form keys) <<<**
<img width="405" height="823"
alt="8-partitions-freeform-default-light-panel"
src="https://github.com/user-attachments/assets/bcba8a73-a7e9-4df0-b334-931c1a195f9f"
/>
## How status is derived
For each key, `E` is the latest event and `R` the newest attempt among the
runs of Dags that declare the asset as an outlet. A run belongs to its own
`partition_key`.
- `R` queued or running, and no `E` since the attempt began: **In Progress**
- `R` failed, and no `E` since the attempt began: **Failed**
- otherwise, `E` exists: **Materialized**
- otherwise the key is not listed
An attempt begins at `queued_at`, which clearing resets. Runs and events of
Dags the caller may not read are left out before aggregation, so neither rows
nor counts reveal them.
## Known limits
- There is no "missing" state: an asset does not declare its partitions, so
an absent key cannot be told from a key that never existed.
- Attribution is per Dag run, not per task. A run that fails before emitting
anything is tied to its own key, even if its task would have emitted another
key. A run that emitted the asset only under other keys is left out.
- Each request scans the asset's event keys and its producer Dags' runs, for
the counts and again for the page. `dag_run.partition_key` has no index.
## Testing
- `uv run --project airflow-core pytest
airflow-core/tests/unit/api_fastapi/core_api/routes/ui/test_assets.py`: 71
passed, 30 of them for the new route.
- `pnpm lint`: clean. `pnpm test --maxWorkers=6`: 2,450 passed. With the
default worker count, three test files unrelated to this change fail on my
machine, and they fail the same way on the base commit.
- `prek run --from-ref main --stage pre-commit`: passed.
- Run in `breeze start-airflow` with a daily partitioned producer and seeded
runs; checked in light and dark mode, at the default and a widened details
panel.
Not run: the manual prek stage, `breeze verify`, PostgreSQL and MySQL
backends, and the Playwright e2e suite.
---
##### Was generative AI tooling used to co-author this PR?
- [X] Yes — Claude Code (Fable 5.2)
Generated-by: Claude Code (Fable 5.2) following [the
guidelines](https://github.com/apache/airflow/blob/main/contributing-docs/05_pull_requests.rst#gen-ai-assisted-contributions)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]