cmarteepants commented on PR #68517: URL: https://github.com/apache/airflow/pull/68517#issuecomment-5702054976
Thanks for picking this up @renat-sagut ! Before this lands, I started a dev-list discussion on what the default should be, advocating that we should NOT be batching by default, even if it would technically be a breaking change. The thread came out consistently in favour of **not** batching by default (1 event -> 1 DagRun): - **Jake McGrath**: sees a real use case for batching, but leans toward not batching asset events by default. - **Jarek Potiuk**: +1 on not batching, and would deprecate/remove auto-batching entirely — the batch size is unpredictable and uncontrollable, so unrelated load on a deployment can silently change a Dag's behavior. - **Philippe Gagnon**: agrees the behavior is counter-intuitive. - **Kaxil Naik**: had assumed it was a bug because it's hard to reason about, and thinks we should change it. Defaulting to today's behaviour looks like the safer choice, but batch size already varies with scheduler cadence and parallelism, so nobody can be relying on a specific value, and it's very misunderstood. The cost of unbatching is an occasional extra DagRun for a few users while the cost of keeping it is everyone else either has to deal with extra events and break their pipelines if not accounted for (which happens a lot since it's very misunderstood as to how this works). Would you be willing to flip the default so that batching is opt-in rather than opt-out? Dev list email chain: https://lists.apache.org/thread/jz98dor8h02mjoxddtdp1cvml9d8yxcr -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
