ieCecchetti commented on issue #63374: URL: https://github.com/apache/airflow/issues/63374#issuecomment-6213218093
Not reproducible with the things tried. Using the Dag from the issue description (callback writes a marker file, so it does not depend on logs being captured), the Dag-level `on_failure_callback` fired every time: | Airflow | Setup | Plain function callback | Notifier (`BaseNotifier` subclass) | |---|---|---|---| | 3.1.7 | `airflow standalone`, Dags folder | fired | – | | 3.2.2 | `airflow standalone`, Dags folder | fired | fired | | 3.2.2 | `airflow standalone`, `GitDagBundle` | fired | – | | 3.2.2 | Helm chart, KubernetesExecutor, separate scheduler / Dag processor pods | fired | fired | | 3.3.2 | `airflow standalone`, Dags folder | fired | fired | Note: the `callback is empty` debug line is logged for every Dag run that has no callback to send (including running ones), so it is not by itself evidence that the callback was dropped. When a Dag callback runs, the Dag processor logs `Executing on_failure dag callback`; if the callback raises, the error is only visible in the Dag processor logs. It may be fixed by these, merged after 3.1.7, each of which fixes a case where a Dag callback was dropped without any log: - #63795 (3.2.0): callbacks starved when several Dag bundles are configured - #66484 (3.3.0): queued callback purged by the periodic file scan with versioned (git) bundles - #70990 (3.3.2): one failing callback request crashing the Dag processor and dropping the rest of the batch - #70987 (3.3.2): callbacks dropped when `dag_version_inflation_check_level` is `error` - #72930 (3.3.3, not released yet): callbacks dropped for git bundles with `disable_bundle_versioning` Could the reporters retest on 3.3.2 and, if it still fails, share the bundle type (Dags folder, git-sync, `GitDagBundle`), whether bundle versioning is disabled, the number of bundles and Dag processor replicas, and the Dag processor logs around the time the run failed? I have not verified which of the fixes above applies to the original setups. --- Drafted-by: Claude Code (Opus 5.5); reviewed by @ieCecchetti before posting -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
