GitHub user hnomichith created a discussion: Add the ability to backfill a DAG
based on past Asset Events
Hello!
# Airflow version
Airflow 3.1.4
# Context
I'm currently working with DAG scheduling on Asset Events. I saw that if you
create a DAG which triggers on an Asset Event, there's no usable way to
backfill it based on past Asset Events. For example, if those Asset Events were
created when the DAG was not created yet, or if the DAG was paused.
I saw two work-around, both with their caveats:
1. Backfill the upstream DAG to re-trigger the events. It's a single operation,
however it implies re-running the upstream DAG when it may not be useful.
2. Manually re-create the asset events via "Assets" -> "select Asset" ->
"Create Event" -> "Manual". However:
- Here you create the events one by one, which is not scalable.
- If you want to "re-publish" an Asset Event, you need to copy its extra
info, then paste them in the new Asset Event.
- If your Asset has several downstream DAGs, you'll run them all, while
you may only want to backfill one.
I identified one more usable solution creating a script to automate the second
solution. It makes it scalable, but still have the caveat of running all
downstream DAGs.
# Questions
I'm opening the discussion to ask some questions:
- Is it a very specific need, or is there some other people who share it?
- Am I mis-using Airflow in a way?
# Potential solution
Instinctively, the solution I see to address my need is:
1. Create a POST `/api/v2/dags/:dag_id/assets/queuedEvents` endpoint
2. Modify GET `/api/v2/assets/events` to add a `name` filter (and everything
that could be used to schedule a DAG based on an Asset)
3. In Airflow UI, from an Asset-scheduled DAG, when clicking on "Trigger" ->
"Backfill", call the endpoint modified in step 2 to show the number of runs to
be triggered ; and on validation call the endpoint modified in step 1.
What do you think?
GitHub link: https://github.com/apache/airflow/discussions/59886
----
This is an automatically sent email for [email protected].
To unsubscribe, please send an email to: [email protected]
