Hi, I have dags with tasks that use same configuration and same schedule time frequencies.
When I have x tasks in a dag they run in //. My dag is scheduled every 15 minutes, but sometimes I have at least a task that runs for more than 15 minutes and 2 identical tasks should not run at the same time (in my case) but it's what happens. So when the dag is kicked off (every 15 min) I need Airflow to run tasks only for those that are not running, which means : - if at time t1 tasks tk1 is running and tk2, ..., tkn are not running, I need Airflow to run only tk2,.., tkn but not tk1 cause it's already running - if at time t2 tasks tk1, ..., tkn are not running, I need Airflow to run tk1,.., tkn I already tried using depends_on_past=True, however I need failed tasks to be kicked off, cause I can get some temporary issues. I also use an upstream task connected to all tasks in my dag to be able to run manually all tasks if I need to. (I don't use a frequency of 15 minutes in all dags) Any idea ?
