khmelevskiy commented on PR #73433: URL: https://github.com/apache/airflow/pull/73433#issuecomment-6061475173
We're hitting this on 3.3.2 in production: standalone dag-processors (one per team), `GitDagBundle`, `min_file_process_interval = 120`. Within two and a half days of a redeploy it started on four files across three dag-processors. Each looping file is parsed 2-3 times a second instead of every 120 s, costs 1-1.5 CPU cores, and its child-process log grows by 80-280 MB a day. A restart clears it; the file's next callback brings it back, usually within a day. One addition to the issue: here the trigger was task-level callbacks, not Dag-level ones. These are the `TaskCallbackRequest`s the scheduler sends for tasks it fails itself. The loop started in the same minute as the file's first callback. The request carries `bundle_version` either way, so this PR covers both. main still resolves the stats by insertion order in `prepare_file_queue` and `_sort_by_mtime`. @jedcunningham @ephraimbuddy could you take a look? A backport to v3-3-test would help too. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
