Yeah, I actually thought this was a bug since it is difficult to reason about!
We should change that On Mon, 14 Sept 2026 at 18:46, Philippe Gagnon via dev < [email protected]> wrote: > I agree. The batching behavior is counter-intuitive. > *✨ **Philippe Gagnon* > *Meet for 30 mins 📅* <https://calendar.app.google/5qzgD9SadybUvSCv8> > > > On Sat, Sep 12, 2026 at 7:26 AM Jarek Potiuk <[email protected]> wrote: > > > +1. Not batching. I think we should generally deprecate / remove the > > auto-batching behavior as it is impossible to predict or control. If we > > introduce batching, it should be configurable: specifying the number of > > messages, the batching time windows etc. Unpredictable behaviour like > this > > is dangerous, especially if users start relying on it and it suddenly > gets > > impacted by unrelated things. In this case the big issue is that many > > "noisy neighbour" cases can change the behaviour in unpredictable ways. > You > > might not even see the batching happening regularly - if events are > > "slow-ish," - but suddenly you might suddenly start working when you > > trigger a backfill for a completely different DAG, for example. That's a > > recipe for disaster. > > > > On Sat, Sep 12, 2026 at 1:20 PM Jarek Potiuk <[email protected]> wrote: > > > > > NOTE: To Jake and others posting to devlist - as not everyone might be > > > aware—please subscribe to devlist before posting (see "community" in > > > https://airflow.apache.org). > > > > > > Otherwise we have to moderate your posts (only subscribers can write > > > without moderation), and you might not see the answer when someone > > replies > > > to "dev@". > > > > > > J. > > > > > > On Fri, Sep 11, 2026 at 6:55 PM Jake McGrath via dev < > > > [email protected]> wrote: > > > > > >> I agree with Constance that this is something that needs a bit more > > >> discussion; this is something that I’ve bumped into a good bit lately, > > >> specifically, with Asset-watching of S3. > > >> > > >> I like to think about this starting with the DAG. If my DAG has Tasks > > >> within it that are only meant to parse a single file, then a single > run > > >> for > > >> > 1 Asset Event would most likely break my DAG/not be properly > handled. > > >> > > >> I do think there is a use-case for the existing “batching” sort of > > >> behavior. However, I’d personally lean towards NOT batching Asset > > Events > > >> by default. > > >> > > >> > > >> > > >> > > >> On Sep 10, 2026 at 7:40:29 PM, Constance Martineau via dev < > > >> [email protected]> wrote: > > >> > > >> > Hi all, > > >> > > > >> > Raising something that has been coming up over the past few years, > > >> > especially recently with Asset Watchers: When a Dag is > > asset-triggered, > > >> the > > >> > scheduler consumes every pending AssetEvent for that Dag in one loop > > and > > >> > creates a single DagRun for all of them > > >> (_create_dag_runs_asset_triggered). > > >> > If five files land and generate five events before the next > scheduler > > >> loop > > >> > runs, you get one DagRun that consumes all five events, not five > > >> separate > > >> > DagRuns. Most users assume 1 event -> 1 DagRun and read the > batchings > > as > > >> > Airflow dropping or missing events, and how much gets batched > depends > > on > > >> > scheduler loop and Dag parallelism settings, not anything declared > in > > >> the > > >> > Dag (see https://github.com/apache/airflow/issues/56750). This > > >> behavior is > > >> > intentional, dating back to when datasets shipped in 2.4, but the > > >> > perception that "this is a bug" is real. This issue has come up as a > > >> > specific point at Airflow Summit talks two years running, with > > speakers > > >> > assuming this was Airflow 2 flakiness that had since been fixed. > > >> > > > >> > It's not just a perception problem, either. Any asset-triggered Dag > > that > > >> > isn't explicitly written to iterate `dag_run.consumed_asset_events` > > will > > >> > silently under-process when multiple events land in the same run. It > > >> looks > > >> > like "1 run = 1 file" and quietly drops the rest. This isn't a > > >> hypothetical > > >> > scenario. The exact use-case that prompted this is that someone in > one > > >> of > > >> > our client facing teams is building an S3 trigger on the > AssetWatcher > > >> > framework (there's no official S3 event trigger for this yet, the > only > > >> > documented AssetWatcher pattern today is SQS) that emits one event > per > > >> > updated file, with several files landing in the same scheduler loop. > > >> > Because how many events get bundled depends on scheduler cadence and > > >> > parallelism rather than anything declared in the Dag, this behavior > is > > >> also > > >> > untestable. There's no way to write a CI test that reliably asserts > "N > > >> > events produce N runs". > > >> > > > >> > There's already community momentum here: > > >> > > > >> > - #55956 <https://github.com/apache/airflow/issues/55956> > proposes > > a > > >> > `max_asset_events` param, milestoned for 3.4.0 > > >> > - #56750 > > >> > < > > >> > > > >> > > > https://medium.com/@MarinAgli1/a-look-into-airflow-data-aware-scheduling-and-dynamic-task-mapping-8c548d4ad79 > > >> > > > > >> > groups the related issues (#53896 > > >> > <https://github.com/apache/airflow/issues/53896>, #56691 > > >> > <https://github.com/apache/airflow/issues/56691>, #56050 > > >> > <https://github.com/apache/airflow/issues/56050> and #47398 > > >> > <https://github.com/apache/airflow/issues/47398>) and proposes a > > >> > Dag-level toggle (`asset_grouping`) rather than an all-or-nothing > > >> global > > >> > switch. > > >> > > > >> > Regarding the question of whether this can only be a global config: > I > > >> don't > > >> > think so. `_create_dag_runs_asset_triggered` already loops per-Dag > and > > >> only > > >> > fetches that Dag's own pending events before building its DagRun, > so a > > >> > per-Dag opt-in is localized to that branch and shouldn't require > > >> touching > > >> > the shared code path every other asset-scheduled Dag depends on. I'd > > >> model > > >> > this the same way we already handle `catchup`: a Dag-level parameter > > >> (like > > >> > `asset_grouping`) that falls back to a global > > >> `asset_grouping_by_default` > > >> > in airflow configs when unset, mirroring `catchup` / > > >> `catchup_by_default`. > > >> > That gives Dag authors an explicit override where usage is genuinely > > >> mixed, > > >> > while still letting an org flip the behaviour fleet-wide for every > Dag > > >> that > > >> > hasn't opted in, without touching Dag code. SCrocky's issue for > > example > > >> > Idescribes running both patterns side by side in the same > deployment. > > >> > > > >> > Where I want actual discussion: What should > > `asset_grouping_by_default` > > >> > ship as? Every proposal so far (including ours) assumes it has to > > >> default > > >> > to today's batched behaviour, to avoid a breaking change. I want to > > make > > >> > the case for defaulting it to `False` (1 event -> 1 DagRun) instead, > > >> > because the cost asymmetric. > > >> > > > >> > - If we default to unbatched and someone was relying on batching, > > the > > >> > worst case is an extra DagRun. They were never able to control the > > >> batch > > >> > size to begin with as it depends on the scheduler cadence and > > >> > parallelism, > > >> > not anything declared in the Dag, so their code already has to > > >> tolerate a > > >> > variable number of events per run. An occasional extra run is just > > >> more > > >> > of > > >> > the same variance they already had to handle, not a new failure > > mode. > > >> > - If we keep batching as the default, everyone who wants per-event > > >> > semantics, which judging by the summit and this thread is most > > >> people's > > >> > mental model, has to actively work around it. They need to inspect > > how > > >> > many > > >> > events landed in a run and add conditional/expansion logic to > split > > >> them > > >> > back out, which is exactly the workaround we're discussing > > internally > > >> > right > > >> > now. This imposes real, ongoing complexity on the majority to > > protect > > >> a > > >> > minority's default. > > >> > > > >> > I don't think this is a close call. > > >> > > > >> > I'm not trying to let perfect be the enemy of good. If we can only > get > > >> > consensus on an opt-in with today's default, that's still a real > > >> > improvement, but I'd rather we make the case for the better default > > >> before > > >> > settling for that. > > >> > > > >> > Curious what others thing, especially anyone closer to #56750 > > >> > < > > >> > > > >> > > > https://medium.com/@MarinAgli1/a-look-into-airflow-data-aware-scheduling-and-dynamic-task-mapping-8c548d4ad79 > > >> > > > > >> > . > > >> > > > >> > Thanks, > > >> > Constance > > >> > > > >> > > > >> > -- > > >> > > > >> > Constance Martineau > > >> > > > >> > Staff Product Manager > > >> > > > >> > Email: [email protected] > > >> > > > >> > Time zone: US Eastern (EST UTC-5 / EDT UTC-4) > > >> > > > >> > > > > > >
