Hi All,
I am having a few issues with Airflow and task instances greater than 750. I
am getting one consistent error and one error that happens random (understand,
it's technically not random).
Consistent error:
[2020-09-25 12:28:01,703] {scheduler_job.py:237} WARNING - Killing PID 119970
[2020-09-25 12:29:17,110] {scheduler_job.py:237} WARNING - Killing PID 121013
[2020-09-25 12:29:17,110] {scheduler_job.py:237} WARNING - Killing PID 121013
[2020-09-25 12:30:12,171] {scheduler_job.py:237} WARNING - Killing PID 123243
Random error:
[2020-09-27 19:37:25,127] {scheduler_job.py:771} INFO - Examining DAG run
<DagRun tutorial_large_design_debug7 @ 2020-09-28 02:37:24+00:00:
manual__2020-09-28T02:37:24+00:00, externally triggered: True> [2020-09-27
19:37:26,749] {logging_mixin.py:112} INFO - [2020-09-27 19:37:26,749]
{dagrun.py:408} INFO - (MySQLdb.exceptions.IntegrityError) (1062, "Duplicate
entry 'echo__a-tutorial_large_design_debug7-2020- 09-28 02:37:24.000000' for
key 'PRIMARY'") [SQL: INSERT INTO task_instance (task_id, dag_id,
execution_date, start_date, end_date, duration, state, try_number, max_tries,
hostname, unixname, job_id, pool, pool_slots, queue, priority_weight, operator,
qu eued_dttm, pid, executor_config) VALUES (%s, %s, %s, %s, %s, %s, %s, %s, %s,
%s, %s, %s, %s, %s, %s, %s, %s, %s, %s, %s)]
Please help! thanks!
