Sooo I was throwing some ideas around with David in a DM, and here are the ones I came up with
some_task.batch(num_tasks=5)? some_task.batch(across=5) ? some_task.spread(across=5) ? some_task.parallelise(across=5)? some_task.distribute(across=5)? some_task.distribute(across_n_tasks=5)? I think I like the `.parallelise(across=n)` the most? It’s not perfect I admit. Also (perhaps oddly) I think even with that I think the feature should be called "Batched Mapped Task” even if the fn is .parallelise or .distribute — but I don’t think "Task Spreading” is very clear from a user-reading-the-docs PoV. -a > On 24 Sep 2026, at 09:51, Amogh Desai <[email protected]> wrote: > > Late to the party, but thanks, David. > > Out of all the options, I vote +1 for `.spread(across=N)`. > > -1 to unfold because, to me, it means generating a sequence from something, > which describes > creating items rather than distributing them. > > And regardless of the name, the round-robin distribution needs to be > explicit in the SDK docs with the reasoning, > as Constance said. The name fixes the "how many" confusion, not the "which > items go where" one. > > Thanks & Regards, > Amogh Desai > > > On Thu, Sep 24, 2026 at 1:48 AM Jarek Potiuk <[email protected]> wrote: > >> How about "scatter" or "unfold" :)? >> >> (just had a little conversation with Claude about it). >> >> On Wed, Sep 23, 2026 at 12:20 PM Blain David <[email protected]> >> wrote: >> >>> I've updated the AIP-104 confluence page and choose the spread option and >>> thus the Task Spreading instead of Batched Task Mapping. >>> >>> Does everyone agree with this change? >>> >>> >>> >> https://cwiki.apache.org/confluence/spaces/AIRFLOW/pages/421954527/AIP-104+Iterable+Tasks+IT+and+Task+Spreading+TS >>> >>> If so, then I will push those changes in the PR as well: >>> >>> https://github.com/apache/airflow/pull/62922 >>> >>> General (Internal Property) >>> ________________________________ >>> From: Shahar Epstein <[email protected]> >>> Sent: Saturday, September 19, 2026 10:37 >>> To: [email protected] <[email protected]> >>> Subject: Re: [DISCUSS] Naming of `.batch(size=)` in AIP-104 (Iterable >>> Tasks and Dynamic Task Batching) >>> >>> EXTERNAL MAIL: Indien je de afzender van deze e-mail niet kent en deze >>> niet vertrouwt, klik niet op een link of open geen bijlages. Bij twijfel, >>> stuur deze e-mail als bijlage naar [email protected]<mailto: >>> [email protected]>. >>> >>> I'd go with .spread(...), as both batch(...) and shard(...) are already >>> semantically overloaded. >>> >>> >>> Shahar >>> >>> On Thu, Sep 17, 2026 at 12:26 PM Blain David <[email protected]> >>> wrote: >>> >>>> Hi all, >>>> >>>> While reviewing the AIP-104 PR ( >>>> >>> >> https://eur01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fapache%2Fairflow%2Fpull%2F62922&data=05%7C02%7Cdavid.blain%40infrabel.be%7Cfb61e6c745c34a77080408df1629466b%7Cb82bc314ab8e4d6fb18946f02e1f27f2%7C0%7C0%7C639254038688014321%7CUnknown%7CTWFpbGZsb3d8eyJFbXB0eU1hcGkiOnRydWUsIlYiOiIwLjAuMDAwMCIsIlAiOiJXaW4zMiIsIkFOIjoiTWFpbCIsIldUIjoyfQ%3D%3D%7C0%7C%7C%7C&sdata=S4qPFkVdn2y1F%2FtLRWhlkqwazsOUbNVrPEi%2FYuHoN7w%3D&reserved=0 >>> )<https://github.com/apache/airflow/pull/62922>, Amogh raised a naming >>>> concern about the Dynamic Task Batching API that I would like to settle >>>> here before the PR merges, >>>> since renaming afterwards would be a breaking change. >>>> >>>> ## What `.batch(size=N)` does today >>>> >>>> `task.batch(size=17).iterate(url=urls)` creates 17 mapped task >> instances >>>> through Dynamic Task >>>> Mapping, and each of them iterates over its share of `urls` in a single >>>> task instance using >>>> Iterable Tasks. The items are distributed round robin: item `i` goes to >>>> task instance `i % 17`. >>>> So `size` is the number of task instances, not a chunk length, and the >>>> task instances do not >>>> receive contiguous slices of the input. >>>> >>>> The round robin scheme is deliberate. The number of task instances has >> to >>>> be fixed before the >>>> underlying iterable is consumed, because the scheduler needs the >> mapping >>>> cardinality up front. >>>> With round robin that number is simply `size`. Contiguous chunking, the >>>> way `itertools.batched` >>>> works, would need `ceil(len(items) / size)` task instances, which is >>>> unknowable until a possibly >>>> unbounded or paginated iterable has been fully drained. That would >> defeat >>>> the purpose of iterating >>>> lazily. >>>> >>>> ## The concern >>>> >>>> The name reads like `itertools.batched(iterable, size)`, where `size` >> is >>>> the chunk length and the >>>> number of chunks is derived from it. Ours is the opposite: the number >> of >>>> task instances is given, >>>> and the per-instance share is derived. The class docstring and the Task >>>> SDK docs currently spend a >>>> full paragraph undoing that first impression. A name that carries the >>>> meaning directly would not >>>> need it. >>>> >>>> ## Options >>>> >>>> 1. Keep `.batch(size=N)` and rely on documentation. Lowest churn, but >> the >>>> mismatch with >>>> `itertools.batched` stays and every new reader has to be corrected. >>>> >>>> 2. Keep the method, rename the parameter: `.batch(count=N)` or >>>> `.batch(tasks=N)`. Keeps the >>>> "Dynamic Task Batching" vocabulary from the AIP, and `count` or >>> `tasks` >>>> says what the number >>>> is. `.batch(tasks=17).iterate(url=urls)` reads as "spread over 17 >>>> tasks, then iterate". >>>> >>>> 3. Rename the method as well, to something that describes partitioning >>>> rather than chunking, for >>>> example `.partition(count=N)`, `.shard(count=N)` or >>>> `.spread(across=N)`. Most descriptive, but >>>> drifts from the AIP's own terminology, and "shard" and "partition" >>>> carry data-engineering >>>> connotations that may suggest contiguous ranges just as much as >>> "batch" >>>> does. >>>> >>>> My preference is option 2 with `count=`. It keeps the API shape and the >>>> AIP terminology, fixes >>>> the misleading part, and costs nothing now because the feature is >>>> unreleased. `size` would be >>>> kept as a rejected alias raising a clear error rather than silently >>>> accepted, so nobody copies >>>> the old spelling from an early draft. >>>> >>>> Unless there are objections or a better name comes up, I will treat >> this >>>> as lazy consensus in >>>> 72 hours and update the PR, the docs and the AIP page accordingly. >>>> >>>> Thanks, >>>> David >>>> >>>> General (Internal Property) >>>> >>> >> --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
