GitHub user hnomichith added a comment to the discussion: Add the ability to backfill a DAG based on past Asset Events
> There is no new APIs needed for that. You can just trigger asset event > manually. > https://airflow.apache.org/docs/apache-airflow/stable/stable-rest-api-ref.html#operation/create_asset_event > -> if Dag2 and Dag3 are unpaused and you want to trigger it "as if the dag1 > did it", you just have to create asset event - exactly the same as Dag 1 > would do. Dag 1 - when completing simply creates the asset event that is it's > output. You can do the same via API. You just needed to know what asset to > create, which extras it should have. Once you create it Dag 2 And Dag 3 will > be triggered. > > You just need to know what assets and with what assets and with parameter you > should trigger. And - as I repeated several times - you just need to (best) > write a script that would find out (according to your criteria) which asset > events and with which parameters you should trigger. > > This is what you can do now and what I think makes sense. That's basically what I described in my very first message, with the automation I mentioned. Actually, the caveat I mentionned can also be worked-around: to trigger DAG_2 or DAG_3 independently and avoid potentially unecessary processes, the script must also pause all DAGs that doesn't need to be triggered, and un-pause them at the end. Given the rest of your answer, having a more integrated solution, either via an API to put an existing event in the DAG queue, or an UI to facilitate it, would raise many more concerns especially for use cases with complexe triggering conditions. Thanks for all you anwers. I'll go for the script, and I'm satisfied meanwhile to have shared the need and assessed that I couldn't easily contribute to Airflow to address it in a more integrated way. Maybe last question for you: is my usecase so exotic? I'm surprised that the need of processing data updates that happened when a DAG was not created yet is not something common. If it's exotic, then it might hide a mis-usage of Airflow on my side? GitHub link: https://github.com/apache/airflow/discussions/59886#discussioncomment-15389690 ---- This is an automatically sent email for [email protected]. To unsubscribe, please send an email to: [email protected]
