seanghaeli commented on issue #76997:
URL: https://github.com/apache/airflow/issues/76997#issuecomment-6861547293
The mental model I'm using for Asset Deadlines (as compared to Dag
Deadlines): both work out a deadline time and check something when it arrives.
A Dag run deadline checks whether the run finished; an asset deadline checks
whether an event landed in the window leading up to it ("by 5am, has there been
an event since midnight?"). So it can reuse the Deadline Alerts machinery with
a new reference and a new check.
<img width="800" alt="A Dag run deadline measures forward from when the run
was queued; an asset deadline checks whether an event landed in the window
before the deadline"
src="https://raw.githubusercontent.com/seanghaeli/airflow/db20bec8ee21142166ffa3f601b35657cdc880b8/issue-76997/asset-deadline-model.png";
/>
Some design questions to mull over, I'll be thinking about these over the
next week:
- **References.** Two new references cover most cases: a schedule ("by 5am
every day, any event since midnight") and the asset's last event ("within 6
hours of the previous one"). Starting with these and leaving room for more
seems safest.
- **Recovery.** Once an asset misses its deadline, should it show as late in
the UI and API until the next event arrives, and should that event trigger a
second callback, so whoever was paged knows it's resolved? I lean towards yes
to both.
- **Who owns a deadline.** The issue's "every declaration is its own
deadline" seems right to me: each declaring Dag keeps its own threshold,
callback and team, since one consumer may need it by 2am and another by 9am.
The main thing this costs is that every Dag that imports the same asset
definition adds a duplicate, but keeping per-consumer flexibility makes it the
better option in my opinion. When several Dags share the exact same deadline on
an asset, they could be pooled: checked once and alerted once, instead of
repeating the work. That's a narrower case than throttling repeated alerts
(#63856), but they might end up sharing a mechanism.
- **Partitions.** For per-key checks, the expected keys can come from the
producer's partition timetable, as the issue suggests. What should we expect
for assets updated by an asset watcher, which have no producer task?
- **Deadlines on a run that hasn't happened.** The issue's `run_deadline`
(for `orders_daily & customers_daily`, "I should have a run for key K by X") is
about the run, not any single input. Copying it onto each input would alert per
input instead of on the missing run, and with `|` it would alert on inputs the
run didn't need.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
![https://raw.githubusercontent.com/seanghaeli/airflow/db20bec8ee21142166ffa3f601b35657cdc880b8/issue-76997/asset-deadline-model.png"](https://raw.githubusercontent.com/seanghaeli/airflow/db20bec8ee21142166ffa3f601b35657cdc880b8/issue-76997/asset-deadline-model.png"){kind=link}