The below is a more organized writeup of a suggestion I once shared on
Slack. I figured it should be documented and discussed here as well. I am
currently developing something equivalent using a short-circuit operator,
but it would be nice for this to be a built-in functionality. An
illustration of the proposed behavior compared to ANY and ALL is provided
as code as well as a rendered image.
Please let me know: 1) what you think; 2) whether this fits into the bigger
vision of Asset-based triggering; and 3) would you be willing to
participate in the implementation of this feature.
So without further ado....
--------------------------------------------------
A "freshness" check is a condition applied to asset updates of multiple
URIs to determine if they're sufficiently new in the context of each other.
Freshness can be used to define temporal constraints between Assets that
are updated at different frequencies. As a result, a freshness check
results in a behavior that's intermediate between AssetAny and AssetAll:
- The check is triggered when ANY of the assets is updated.
- The check is satisfied when ALL of the assets are *sufficiently* new
("fresh").
```none
Illustration (+ represents a trigger, | represents an interval boundary):
Intrvl 1 Intrvl 2 Intrvl 3 Intrvl 4
Asset A: |----1----|--2------|------3--|----------|
Asset B: |---a--b--|---c--d--|--e-f----|---------g|
All: |----+-------+-------------+-------------| Times triggered: 3
Any: |---++-+-----++--+-----+-+-+------------+| Times triggered: 10
Fresh: |----+-+--|---+--+--|------+--|----------| Times triggered: 5
```
Thus, within each interval, freshness initially behaves like ALL, but
"becomes" ANY after the condition is met.
Claude was easily able to suggest potential use cases:
- Data Drift Detection: Comparing week-old predictions against fresh
labels (or vice versa) produces misleading drift calculations.
- A/B Test Analysis: Update analysis only when both groups have data
from the same time window, since temporal misalignment between groups can
create false positive/negative results due to seasonality or external
events.
- Real-time vs Batch Consistency Monitoring: Flag inconsistencies only
when both pipelines have processed the same time range.
- Data Quality Monitoring: Calculate data quality metrics only when
processing lag is under acceptable threshold.
[image: image.png]
