[GitHub] [airflow] avkirilishin commented on pull request #21214: Removes the next_dagrun_create_after reset

[GitBox]

avkirilishin commented on pull request #21214:
URL: https://github.com/apache/airflow/pull/21214#issuecomment-1030277109


   > but it didn't resolve the linked issue
   
   This fix resolves the linked issue. The new test 
`test_runs_are_created_after_max_active_runs_was_reached` verifies this.
   
   I have to explain how it works:
   1. The value of `max_active_runs` has been reached, and the value of 
`next_dagrun_create_after` has been reset: 
https://github.com/apache/airflow/blob/ab762a5a8ae147ae33500ee3c7e7a73d25d03ad7/airflow/jobs/scheduler_job.py#L975
   2. Due to filtering `next_dagrun_create_after`, 
`DagModel.dags_needing_dagruns` (referred to in 
`SchedulerJob._create_dagruns_for_dags`) cannot select Dags to create new runs. 
Comparison predicates with NULL values always return UNKNOWN (==FALSE in that 
case).
   
https://github.com/apache/airflow/blob/ab762a5a8ae147ae33500ee3c7e7a73d25d03ad7/airflow/models/dag.py#L2866-L2876
   3. Dag sticks and runs are no longer being created
   4. If `AIRFLOW__SCHEDULER__MIN_FILE_PROCESS_INTERVAL` is low, the scheduler 
processes dag's file and calls `Dag.bulk_write_to_db`, which calls 
`calculate_dagrun_date_fields` and calculates the new 
`next_dagrun_create_after` value
   
https://github.com/apache/airflow/blob/ab762a5a8ae147ae33500ee3c7e7a73d25d03ad7/airflow/models/dag.py#L2441-L2444
   5. But if `AIRFLOW__SCHEDULER__MIN_FILE_PROCESS_INTERVAL` is high, nobody 
calculates `next_dagrun_create_after`
   6. This merge request corrects situation by removing the 
`next_dagrun_create_after` nullify
   
   ---
   
   `next_dagrun_create_after` has been reset:
   
   <img width="920" alt="Screenshot 2022-02-04 at 21 51 39" 
src="https://user-images.githubusercontent.com/54231417/152586591-3d02a5ce-5873-4982-a3ba-2feb27fd0c98.png";>
   
   `min_file_process_interval` is 180. After 3 minutes (16:21:36 - 16:24:37), 
the scheduler processes dag's file and calculates the new 
`next_dagrun_create_after` value:
   
   <img width="920" alt="Screenshot 2022-02-04 at 21 56 18" 
src="https://user-images.githubusercontent.com/54231417/152586987-6e62976e-04aa-4dc1-843f-6370114937c1.png";>
   
   There is a gap in launches:
   
   <img width="1101" alt="image" 
src="https://user-images.githubusercontent.com/54231417/152586855-125fd1d6-6761-4cae-97d9-d8bf856c21f1.png";>
   
   Pay attention to the same `last_parsed_time` and `Queued At` in the last two 
screenshots.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org