Lee-W commented on issue #59294:
URL: https://github.com/apache/airflow/issues/59294#issuecomment-3950237603
A few concrete scenarios for `to_downstream(self) -> str | Iterable[str]`:
* Weekly model training --> daily inference: ML model that's expensive to
train (only needs retraining weekly) but runs inference daily against fresh
data. When weekly partition completes, it needs to enable 7 daily inference
partitions.
* Weekly data deliver --> daily processing: External vendor provides weekly
data files (contractually they only deliver weekly), but business needs daily
processing/reporting. When the weekly file arrives, kick of 7 daily processing
jobs for the week that just completed.
* Weekly aggregations --> daily dashboard: Expensive weekly rollups that get
servedthrough a daily-sliced dashboard or API.
For these to work, `to_downstream("2026-W10")` would need to return
something like: `["2026-03-04", "2026-03-05", "2026-03-06", "2026-03-07",
"2026-03-08", "2026-03-09", "2026-03-10"]`
The tricky part is there are two different temporal relationships:
Backward looking: Weekly W10 (March 4-10) enables 7 daily partitions WITHIN
that week (March 4-10). All 7 would trigger at once when weekly completes.
Forward-looking: Weekly W10 enables the NEXT 7 daily partitions (March
11-17). They trigger according to their own schedule as each data arrives.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
