GitHub user nathadfield created a discussion: Event-driven scheduling: duplicate DAG runs when an `AssetWatcher` runs on multiple triggerers
## The problem When you run more than one triggerer (the default for HA), an `AssetWatcher` that polls some external state seems to fire once *per triggerer* for the same change, so the consumer DAG runs twice. We hit this with a watcher that polls a Git repo's `main` HEAD and triggers a re-index when it advances. On a deployment with two triggerers, every merge kicks off two identical runs of the (fairly expensive) pipeline. Cron and manual runs are fine; only the asset-triggered ones double up, and the number of duplicates tracks the number of triggerers. Digging in, it looks like Airflow dedupes the *watcher definition* (there's a single trigger behind the asset), but not the *events* it emits: each triggerer runs its own copy of the watcher, independently notices the same change, and emits its own event, and each event creates a run. So you effectively get at-least-once delivery with no way to opt into "one run per change." ## How this differs from #54491 This looks similar to #54491 / #63507 (duplicate asset-triggered runs in HA) but it's a different cause. Those are about multiple *schedulers* both turning a **single** event into two runs (fixed with a row-level lock, PR #60773). Here there are genuinely **two events** from two *triggerers*, so that fix doesn't apply. The scheduler-side case is covered; the triggerer-side one doesn't seem to be discussed anywhere. ## What we do about it today The usual "make the consumer idempotent" approach: the DAG's first step skips if it's already processed this particular change, so the second run is a no-op. It works, but every author of an event-driven DAG has to reinvent it, and the duplicate run still gets scheduled before it's skipped. ## Possible directions A few options, roughly least to most invasive: - A **dedupe key on `AssetWatcher`** (derived from the event) so identical events collapse into one run. This feels like the most natural fit. - **Coalescing identical events** on the scheduler side within a short window. - An opt-in **single-runner watcher** (one triggerer owns it, with failover) for cases where you'd rather have exactly-once than zero-gap coverage. This one's more architectural, hence a discussion rather than a bug report. - At the very least, **document** that event delivery is at-least-once with multiple triggerers, so people know to make consumers idempotent. ## Questions - Is this a known/accepted property of event-driven scheduling, or a gap worth closing? - If it's worth closing, is a dedupe key on `AssetWatcher` the right first step, or does the single-runner idea need an AIP? - Is there a way to make a polling watcher exactly-once across triggerers today that we've missed? Happy to help turn the outcome into a feature request, an AIP, or the docs note. Related: #54491, #63507, PR #60773. GitHub link: https://github.com/apache/airflow/discussions/69319 ---- This is an automatically sent email for [email protected]. To unsubscribe, please send an email to: [email protected]
