jens-scheffler-bosch commented on issue #30327: URL: https://github.com/apache/airflow/issues/30327#issuecomment-1700626246
I tested this bug report with the provided example on Airflow 2.7.0 and latest main and can confirm this "strange" behavior. Made some regression and can confirm that even when `catchup=False` is set, an execution of a manual run from a DAG with a custom TimeTable adds scheduled jobs filling the gap. In total with my tests the scheduling behavior looks inconsistent like below: Scheduling Expression | Initial Run of Last Time Period | Filling Gaps | Notes --- | --- | --- | --- Custom TImetable as example `AfterWorkdayTimetable` | No. Current time period is not started automatically | Yes. When a manual with a logical date in the past is executed, occurrences after this are started and fill the gaps. | This is actually unexpected and inconsistent to other default timetables. Classic cron expression `"0 0 * * *"` | Yes. An initial run for the current period is made | No. | This is how I expected it as well for a custom timetable. Usage of `CronTriggerTimetable` | Yes. An initial run for the current period is made | No. | This is how I expected it as well for a custom timetable. Usage of `EventsTimetable` | Ups, all events defined in the past were executed. | n/a | This was a surprise. Whereas I would see this also as a feature but all defined events in the past were run even when `catchup=False` was set. Might be another bug? Or explicity feature? Could not find any mentioning in the docs, so rather documentation flaw of bug. Usage of `datetime.timedelta()` | Yes. An initial run for the current period is made | ? Another ups. A second run was scheduled after the manual trigger. | This was also rather unexpected. Can be repeated. Whenever a run with a logical date in the past is triggered, one additional scheduled run based on current time is started as well. I tried to walk through the code in `airflow/jobs/scheduler_job_runner.py` which I assume is the core part of logic where this is implemented. But after drilling into the code I was lost :-( and not able to pin down the root error. I fear an scheduler expert is needed here for investigation. (@ashb ?) I assume it is an artifact related to the AIP-39 integration. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
