ephraimbuddy commented on pull request #21901: URL: https://github.com/apache/airflow/pull/21901#issuecomment-1060284341
> Hey @ephraimbuddy - any more context on this one? I'd love to understand more context but I am afraid I am missing it :). Ok. The issue is that the scheduler crashes when running more than 2 schedulers. Looking at where this error is coming from, this may also occur with one scheduler but I saw it while running 3 schedulers. What we do currently is to nullify `next_dagrun_create_after` in the scheduler to prevent creating new dagruns when we have reached max_active_runs. https://github.com/apache/airflow/blob/4e05bbc92557f5eafaa37509387b2fb2b0ab3d4a/airflow/jobs/scheduler_job.py#L988 We calculate this field once the dagrun has completed https://github.com/apache/airflow/blob/4e05bbc92557f5eafaa37509387b2fb2b0ab3d4a/airflow/jobs/scheduler_job.py#L1101-L1102 I'm proposing that instead of nullifying and calculating this field, we should have a new column on the DagModel to track the max_active_runs. Because it looks to me that the operation of recalculating this field is expensive. Here's a dag I used to reproduce this behaviour: ```python from airflow.sensors.date_time import DateTimeSensor from airflow import DAG from pendulum import datetime def create_dag(dag_name): dag = DAG( dag_name, schedule_interval='@daily', start_date=datetime(2022, 1, 1), ) check_time = DateTimeSensor( task_id='check_time', target_time=datetime(2022, 3, 8, 12, 0, 0), poke_interval=30, mode='reschedule', dag=dag ) check_time return dag for i in range(100): globals()[f'dag_{i}'] = create_dag(f'dag_{i}') ``` First, you need to have the schedulers running, before triggering all the dags. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
