+1 on the uniqueness of the solution. Wondering how it worked! -s
On Sun, Jul 10, 2016 at 9:02 AM, Cyril Scetbon <[email protected]> wrote: > Interesting. Thanks for this solution Lance, gonna try it > > On Jul 6, 2016, at 19:11, Lance Norskog <[email protected]> wrote: > > > > You could use the XCOM feature to post a semaphore at the start of the > task > > and then remove it at the end. > > Another task would see the semaphore and immediately quit. > > If you get a race condition, the third 15-minute task will take care of > the > > problem, 15 minutes late. > > > > Lance > > > > On Wed, Jul 6, 2016 at 2:50 PM, Cyril Scetbon <[email protected]> > wrote: > > > >> Hi, > >> > >> I have dags with tasks that use same configuration and same schedule > time > >> frequencies. > >> > >> When I have x tasks in a dag they run in //. My dag is scheduled every > 15 > >> minutes, but sometimes I have at least a task that runs for more than 15 > >> minutes and 2 identical tasks should not run at the same time (in my > case) > >> but it's what happens. > >> > >> So when the dag is kicked off (every 15 min) I need Airflow to run tasks > >> only for those that are not running, which means : > >> > >> - if at time t1 tasks tk1 is running and tk2, ..., tkn are not running, > I > >> need Airflow to run only tk2,.., tkn but not tk1 cause it's already > running > >> - if at time t2 tasks tk1, ..., tkn are not running, I need Airflow to > run > >> tk1,.., tkn > >> > >> I already tried using depends_on_past=True, however I need failed tasks > to > >> be kicked off, cause I can get some temporary issues. > >> > >> I also use an upstream task connected to all tasks in my dag to be able > to > >> run manually all tasks if I need to. (I don't use a frequency of 15 > minutes > >> in all dags) > >> > >> Any idea ? > > > > > > > > > > -- > > Lance Norskog > > [email protected] > > Redwood City, CA > >
