cmarteepants opened a new issue, #76997:
URL: https://github.com/apache/airflow/issues/76997
### Problem
There’s no way to get alerted when something didn’t happen in an
asset-driven pipeline. Deadline Alerts are only created when a Dag run is
created, so every reference type assumes a run exists. If the run never shows
up, there’s nothing to miss a deadline.
This happens a lot:
- a producer Dag stops running because it was paused, or a task keeps
failing and never emits an asset event
- a watcher stops firing because it’s misconfigured, or something broke in
the upstream system Airflow is watching
- a partitioned consumer gets stuck waiting on an input that’s never coming
(apache/airflow#76979)
In all of these cases the expected Dag run quietly doesn’t happen. Users are
working around it with watchdog Dags that check for asset events on a cadence,
or by going back to external task sensors with timeouts.
### Proposal
Users should be able to put a deadline on an asset and express things like
“this asset should get an event every N hours” or “by 9am every day.” That
catches every case above. This should extend DeadlineReferences (#44003), so it
reuses the existing callbacks and UI.
Partitioned assets should be in scope, since that’s where the most pain is.
“Last event age” won’t catch one stuck key if the other keys keep arriving, so
for partitions the check is per key: “2026-11-12 hasn’t arrived by X.” For
time-based partitions, the producer’s partition timetable already tells us
which keys to expect and when, so the deadline is relative to the end of each
partition period. The callback gets the missing key(s), so whoever gets paged
knows exactly what to fix.
**Producer vs Consumer**
There are two questions here, asked by different people:
- **Producer**: “Did I deliver on time?” The team that owns the asset sets
it, and the callback goes to them. This is the contract for consumers.
- **Consumer**: “Did I get what I need on time?” Each consumer sets its own
threshold, and the callback goes to their team.
We need both. Consumers of the same asset have different tolerances (one
needs it by 2am, another by 9am), and a consumer shouldn’t depend on the
producer team setting an expectation.
Assets are declared in many Dags, so every declaration is its own deadline,
and the callback goes to whoever declared it. Whether the declaring Dag has the
asset as an outlet or in its schedule tells us if it’s the producer or a
consumer. Watcher-fed assets have no producer task, so the deadline lives on
the asset, not on an outlet.
Two things to handle:
- **Duplicates**: If everyone imports the same asset definition from a
shared module, the same callback shouldn’t fire six times. Identical
declarations (same deadline and callback) count once.
- **Showing the contract**: The Asset UI shows every deadline on an asset,
labelled by the Dag that declared it, with the producer’s marked as the
contract.
**Consumers with more than one input**
When a Dag depends on several assets with different thresholds, each input
carries its own deadline. The consumer also gets one deadline on the run
itself: “I should have a run for key K by X.” The per-asset alerts tell you
which input is late, and the run deadline tells you the outcome.
Example (API shape TBD):
```
# Producer's Dag: the contract
orders_daily = Asset(
"orders_daily",
deadline=AssetDeadline(cron="0 9 * * *", callback=producer_alert),
)
# A consumer's Dag: its own expectation of the same asset
orders_daily = Asset(
"orders_daily",
deadline=AssetDeadline(interval=timedelta(hours=6),
callback=consumer_alert),
)
# Consumer with several inputs: one deadline on the run itself
schedule=PartitionedAssetTimetable(
assets=orders_daily & customers_daily,
run_deadline=AssetDeadline(interval=timedelta(hours=8),
callback=consumer_alert),
)
```
Related: apache/airflow#76979 / #76980 (recovering a stuck partition once
you know about it)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
