Hi , Sandeep , Its not quite clear what you want. But if I understood correctly may be you can try depend_on_past as True or max_active_runs at dag level.
Thanks , Sunil Khaire On Fri, 2 Oct 2020 at 5:32 PM Sandeep S <[email protected]> wrote: > Hi All, > > I am having a production issue running only one instance of DAG at a time. > If the DAG is running one instance, a 2nd instance does not kick off. But > if any of the task fails in the active DAG instance, the DAG gets marked > failed but a 2nd instance kicks off after 5 mins(5 mins scheduled time for > DAG. > > Please help. > > Regards > Sandeep > > On Mon, Sep 28, 2020 at 1:18 PM Tavares Forby <[email protected]> > wrote: > >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> Hi All, >> >> >> >> >> >> >> >> >> >> >> >> I am having a few issues with Airflow and task instances greater than >> 750. I am getting one consistent error and one error that happens random >> (understand, it's technically not random). >> >> >> >> >> >> >> >> >> >> >> >> Consistent error: >> >> >> >> >> [2020-09-25 12:28:01,703] {scheduler_job.py:237} >> >> WARNING - Killing PID 119970 [2020-09-25 12:29:17,110] >> {scheduler_job.py:237} WARNING - Killing PID 121013 [2020-09-25 >> 12:29:17,110] {scheduler_job.py:237} WARNING - Killing PID 121013 >> [2020-09-25 12:30:12,171] {scheduler_job.py:237} WARNING - Killing PID >> >> 123243 >> >> >> >> >> >> >> >> >> >> >> >> >> >> Random error: >> >> >> >> >> [2020-09-27 >> >> 19:37:25,127] {scheduler_job.py:771} INFO - Examining DAG run <DagRun >> tutorial_large_design_debug7 @ 2020-09-28 02:37:24+00:00: >> manual__2020-09-28T02:37:24+00:00, externally triggered: True> [2020-09-27 >> 19:37:26,749] {logging_mixin.py:112} INFO - [2020-09-27 >> >> 19:37:26,749] {dagrun.py:408} INFO - (MySQLdb. >> >> *exceptions.IntegrityError) (1062, "Duplicate entry >> 'echo__a*-tutorial_large_design_debug7-2020- >> 09-28 02:37:24.000000' for key 'PRIMARY'") >> >> [SQL: INSERT INTO task_instance (task_id, dag_id, execution_date, >> start_date, end_date, duration, state, try_number, max_tries, hostname, >> unixname, job_id, pool, pool_slots, queue, priority_weight, operator, qu >> eued_dttm, pid, executor_config) VALUES (%s, >> >> %s, %s, %s, %s, %s, %s, %s, %s, %s, %s, %s, %s, %s, %s, %s, %s, %s, %s, >> %s)] >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> Please >> >> help! thanks! >> >> >> >> >> >> >> > >
