I agree. The batching behavior is counter-intuitive. *✨ **Philippe Gagnon* *Meet for 30 mins 📅* <https://calendar.app.google/5qzgD9SadybUvSCv8>
On Sat, Sep 12, 2026 at 7:26 AM Jarek Potiuk <[email protected]> wrote: > +1. Not batching. I think we should generally deprecate / remove the > auto-batching behavior as it is impossible to predict or control. If we > introduce batching, it should be configurable: specifying the number of > messages, the batching time windows etc. Unpredictable behaviour like this > is dangerous, especially if users start relying on it and it suddenly gets > impacted by unrelated things. In this case the big issue is that many > "noisy neighbour" cases can change the behaviour in unpredictable ways. You > might not even see the batching happening regularly - if events are > "slow-ish," - but suddenly you might suddenly start working when you > trigger a backfill for a completely different DAG, for example. That's a > recipe for disaster. > > On Sat, Sep 12, 2026 at 1:20 PM Jarek Potiuk <[email protected]> wrote: > > > NOTE: To Jake and others posting to devlist - as not everyone might be > > aware—please subscribe to devlist before posting (see "community" in > > https://airflow.apache.org). > > > > Otherwise we have to moderate your posts (only subscribers can write > > without moderation), and you might not see the answer when someone > replies > > to "dev@". > > > > J. > > > > On Fri, Sep 11, 2026 at 6:55 PM Jake McGrath via dev < > > [email protected]> wrote: > > > >> I agree with Constance that this is something that needs a bit more > >> discussion; this is something that I’ve bumped into a good bit lately, > >> specifically, with Asset-watching of S3. > >> > >> I like to think about this starting with the DAG. If my DAG has Tasks > >> within it that are only meant to parse a single file, then a single run > >> for > >> > 1 Asset Event would most likely break my DAG/not be properly handled. > >> > >> I do think there is a use-case for the existing “batching” sort of > >> behavior. However, I’d personally lean towards NOT batching Asset > Events > >> by default. > >> > >> > >> > >> > >> On Sep 10, 2026 at 7:40:29 PM, Constance Martineau via dev < > >> [email protected]> wrote: > >> > >> > Hi all, > >> > > >> > Raising something that has been coming up over the past few years, > >> > especially recently with Asset Watchers: When a Dag is > asset-triggered, > >> the > >> > scheduler consumes every pending AssetEvent for that Dag in one loop > and > >> > creates a single DagRun for all of them > >> (_create_dag_runs_asset_triggered). > >> > If five files land and generate five events before the next scheduler > >> loop > >> > runs, you get one DagRun that consumes all five events, not five > >> separate > >> > DagRuns. Most users assume 1 event -> 1 DagRun and read the batchings > as > >> > Airflow dropping or missing events, and how much gets batched depends > on > >> > scheduler loop and Dag parallelism settings, not anything declared in > >> the > >> > Dag (see https://github.com/apache/airflow/issues/56750). This > >> behavior is > >> > intentional, dating back to when datasets shipped in 2.4, but the > >> > perception that "this is a bug" is real. This issue has come up as a > >> > specific point at Airflow Summit talks two years running, with > speakers > >> > assuming this was Airflow 2 flakiness that had since been fixed. > >> > > >> > It's not just a perception problem, either. Any asset-triggered Dag > that > >> > isn't explicitly written to iterate `dag_run.consumed_asset_events` > will > >> > silently under-process when multiple events land in the same run. It > >> looks > >> > like "1 run = 1 file" and quietly drops the rest. This isn't a > >> hypothetical > >> > scenario. The exact use-case that prompted this is that someone in one > >> of > >> > our client facing teams is building an S3 trigger on the AssetWatcher > >> > framework (there's no official S3 event trigger for this yet, the only > >> > documented AssetWatcher pattern today is SQS) that emits one event per > >> > updated file, with several files landing in the same scheduler loop. > >> > Because how many events get bundled depends on scheduler cadence and > >> > parallelism rather than anything declared in the Dag, this behavior is > >> also > >> > untestable. There's no way to write a CI test that reliably asserts "N > >> > events produce N runs". > >> > > >> > There's already community momentum here: > >> > > >> > - #55956 <https://github.com/apache/airflow/issues/55956> proposes > a > >> > `max_asset_events` param, milestoned for 3.4.0 > >> > - #56750 > >> > < > >> > > >> > https://medium.com/@MarinAgli1/a-look-into-airflow-data-aware-scheduling-and-dynamic-task-mapping-8c548d4ad79 > >> > > > >> > groups the related issues (#53896 > >> > <https://github.com/apache/airflow/issues/53896>, #56691 > >> > <https://github.com/apache/airflow/issues/56691>, #56050 > >> > <https://github.com/apache/airflow/issues/56050> and #47398 > >> > <https://github.com/apache/airflow/issues/47398>) and proposes a > >> > Dag-level toggle (`asset_grouping`) rather than an all-or-nothing > >> global > >> > switch. > >> > > >> > Regarding the question of whether this can only be a global config: I > >> don't > >> > think so. `_create_dag_runs_asset_triggered` already loops per-Dag and > >> only > >> > fetches that Dag's own pending events before building its DagRun, so a > >> > per-Dag opt-in is localized to that branch and shouldn't require > >> touching > >> > the shared code path every other asset-scheduled Dag depends on. I'd > >> model > >> > this the same way we already handle `catchup`: a Dag-level parameter > >> (like > >> > `asset_grouping`) that falls back to a global > >> `asset_grouping_by_default` > >> > in airflow configs when unset, mirroring `catchup` / > >> `catchup_by_default`. > >> > That gives Dag authors an explicit override where usage is genuinely > >> mixed, > >> > while still letting an org flip the behaviour fleet-wide for every Dag > >> that > >> > hasn't opted in, without touching Dag code. SCrocky's issue for > example > >> > Idescribes running both patterns side by side in the same deployment. > >> > > >> > Where I want actual discussion: What should > `asset_grouping_by_default` > >> > ship as? Every proposal so far (including ours) assumes it has to > >> default > >> > to today's batched behaviour, to avoid a breaking change. I want to > make > >> > the case for defaulting it to `False` (1 event -> 1 DagRun) instead, > >> > because the cost asymmetric. > >> > > >> > - If we default to unbatched and someone was relying on batching, > the > >> > worst case is an extra DagRun. They were never able to control the > >> batch > >> > size to begin with as it depends on the scheduler cadence and > >> > parallelism, > >> > not anything declared in the Dag, so their code already has to > >> tolerate a > >> > variable number of events per run. An occasional extra run is just > >> more > >> > of > >> > the same variance they already had to handle, not a new failure > mode. > >> > - If we keep batching as the default, everyone who wants per-event > >> > semantics, which judging by the summit and this thread is most > >> people's > >> > mental model, has to actively work around it. They need to inspect > how > >> > many > >> > events landed in a run and add conditional/expansion logic to split > >> them > >> > back out, which is exactly the workaround we're discussing > internally > >> > right > >> > now. This imposes real, ongoing complexity on the majority to > protect > >> a > >> > minority's default. > >> > > >> > I don't think this is a close call. > >> > > >> > I'm not trying to let perfect be the enemy of good. If we can only get > >> > consensus on an opt-in with today's default, that's still a real > >> > improvement, but I'd rather we make the case for the better default > >> before > >> > settling for that. > >> > > >> > Curious what others thing, especially anyone closer to #56750 > >> > < > >> > > >> > https://medium.com/@MarinAgli1/a-look-into-airflow-data-aware-scheduling-and-dynamic-task-mapping-8c548d4ad79 > >> > > > >> > . > >> > > >> > Thanks, > >> > Constance > >> > > >> > > >> > -- > >> > > >> > Constance Martineau > >> > > >> > Staff Product Manager > >> > > >> > Email: [email protected] > >> > > >> > Time zone: US Eastern (EST UTC-5 / EDT UTC-4) > >> > > >> > > >
