seanghaeli opened a new issue, #77052: URL: https://github.com/apache/airflow/issues/77052
When a trigger fails and something still needs it, like an asset watcher, the triggerer starts it again on its next loop: `update_triggers` (`airflow-core/src/airflow/jobs/triggerer_job_runner.py`) sees an id it should run and isn't running, and re-creates it. A watcher with expired credentials or an unreachable endpoint is re-created about every 2 seconds for as long as it fails, and each failure logs a traceback, counts in `triggers.failed` and writes to the trigger's row (`Trigger.submit_failure`, from #76101). The same happens for: - a watcher whose class can't be imported: `create_triggers` reports `Trigger failed to load code`, and the import is tried again every time, - watchers sharing one stream: when the stream's poll fails, `_SharedStreamGroup._fail_group` (`airflow-core/src/airflow/triggers/shared_stream.py`) fails every subscriber, and they're all re-created together. Suggestion: wait before re-creating a failed trigger, with a delay that grows with its `consecutive_failures` up to a configurable cap. That count is already on the row, so the delay carries over when the trigger moves to another triggerer, and a trigger that keeps failing costs little and writes its row rarely. Give watchers on one shared stream the same delay, so they come back as one group instead of reopening the stream one by one. Deferred tasks' triggers still fail their task right away, as they do now. When adding this, don't count a trigger as recovered (`TriggerFailureTracker.pop_recovered`) while it's being cancelled or released: with #76210, `on_kill()` can keep it running for up to `[triggerer] on_kill_timeout` (30 seconds by default). Done when: a failing trigger waits longer between attempts after each failure, up to the cap, the wait survives a move to another triggerer, and watchers on one shared stream back off together, with tests for each. Related to the triggerer performance work in #76209: a trigger that keeps failing writes its row and reopens its connections on every retry. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
