chinmayasameeru opened a new pull request, #73776: URL: https://github.com/apache/airflow/pull/73776
Fixes #73432 — the dag processor can get stuck in an infinite loop after a Dag-level callback runs. Here's what happens: when a Dag file gets processed in "callback-only" mode (meaning it was triggered by a Dag callback, not a regular scan), `process_parse_results` creates a new `DagFileStat` with `last_finish_time=None`. That stat overwrites the existing one in `_file_stats`, so on the next scan the file looks like it's never been parsed. The scheduler re-queues it, processes it again, creates another stat with `last_finish_time=None`, and the cycle repeats forever. The fix is to preserve the existing `last_finish_time` when doing callback-only processing. I added a `last_finish_time` parameter to `process_parse_results` — when it's passed (from the existing stat), the `is_callback_only` branch uses it instead of the current `finish_time`. I updated both call sites: - `handle_parsing_result`: passes `self._file_stats[file].last_finish_time` — this is the main fix for the reported issue - `process_unparsed_files`: passes `db_stat.last_finish_time` to `DagFileStat.from_db` — this handles the case where a file is already in the DB I couldn't run the full airflow test suite here (the test environment needs a database and a bunch of infra), but I did validate the Python syntax with `py_compile` and the logic is pretty straightforward — it's just passing a timestamp through instead of dropping it. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
