V0lantis commented on issue #23361: URL: https://github.com/apache/airflow/issues/23361#issuecomment-1133120999
I experience exactly the same issue. Reducing the number of schedulers to one seems to have resolved the issue for my deployment, but now, tasks are scheduling a lot slower. I initially had 3 schedulers. Here are my new config variables: ```cfg [scheduler] job_heartbeat_sec = 30 scheduler_heartbeat_sec = 5 num_runs = -1 scheduler_idle_sleep_time = 1 min_file_process_interval = 30 dag_dir_list_interval = 120 print_stats_interval = 240 pool_metrics_interval = 5 scheduler_health_check_threshold = 60 orphaned_tasks_check_interval = 300.0 scheduler_zombie_task_threshold = 300 catchup_by_default = True max_tis_per_query = 512 use_row_level_locking = False max_dagruns_to_create_per_loop = 100 max_dagruns_per_loop_to_schedule = 200 schedule_after_task_execution = True parsing_processes = 2 file_parsing_sort_mode = modified_time use_job_schedule = True allow_trigger_in_future = False dependency_detector = airflow.serialization.serialized_objects.DependencyDetector trigger_timeout_check_interval = 15 run_duration = 41460 ``` -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
