Hi guys, I am trying to build a pipeline/script to monitor our Data-processing pipeline :)
Basically, I am trying to do these things: 1. Go back in time n hours. and Get status of a TASK for last n hours (assuming hourly jobs) I can use the airflow CLI command: "*task_state" * to achieve this. So this tells me where the job has failed/succeeded/running etc. 2. Once I figure out, if some execution of a TASK has state "failed", I want to change the state to "running" again. so that scheduler picks it up and runs it?? *Is there a way to do this? * I think one way to do this is: if a Task is in failed state ---> user "airflow clear" and CLEAR the state. so that scheduler picks it up. But I am not sure how much I can depend on this approach? Will this always work? I just want to think out loud and know if there is a better way to doing this that I am not looking at? Either through code? a new monitoring pipeline? Thanks, Harish
