Late to the party, but thanks, David. Out of all the options, I vote +1 for `.spread(across=N)`.
-1 to unfold because, to me, it means generating a sequence from something, which describes creating items rather than distributing them. And regardless of the name, the round-robin distribution needs to be explicit in the SDK docs with the reasoning, as Constance said. The name fixes the "how many" confusion, not the "which items go where" one. Thanks & Regards, Amogh Desai On Thu, Sep 24, 2026 at 1:48 AM Jarek Potiuk <[email protected]> wrote: > How about "scatter" or "unfold" :)? > > (just had a little conversation with Claude about it). > > On Wed, Sep 23, 2026 at 12:20 PM Blain David <[email protected]> > wrote: > > > I've updated the AIP-104 confluence page and choose the spread option and > > thus the Task Spreading instead of Batched Task Mapping. > > > > Does everyone agree with this change? > > > > > > > https://cwiki.apache.org/confluence/spaces/AIRFLOW/pages/421954527/AIP-104+Iterable+Tasks+IT+and+Task+Spreading+TS > > > > If so, then I will push those changes in the PR as well: > > > > https://github.com/apache/airflow/pull/62922 > > > > General (Internal Property) > > ________________________________ > > From: Shahar Epstein <[email protected]> > > Sent: Saturday, September 19, 2026 10:37 > > To: [email protected] <[email protected]> > > Subject: Re: [DISCUSS] Naming of `.batch(size=)` in AIP-104 (Iterable > > Tasks and Dynamic Task Batching) > > > > EXTERNAL MAIL: Indien je de afzender van deze e-mail niet kent en deze > > niet vertrouwt, klik niet op een link of open geen bijlages. Bij twijfel, > > stuur deze e-mail als bijlage naar [email protected]<mailto: > > [email protected]>. > > > > I'd go with .spread(...), as both batch(...) and shard(...) are already > > semantically overloaded. > > > > > > Shahar > > > > On Thu, Sep 17, 2026 at 12:26 PM Blain David <[email protected]> > > wrote: > > > > > Hi all, > > > > > > While reviewing the AIP-104 PR ( > > > > > > https://eur01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fapache%2Fairflow%2Fpull%2F62922&data=05%7C02%7Cdavid.blain%40infrabel.be%7Cfb61e6c745c34a77080408df1629466b%7Cb82bc314ab8e4d6fb18946f02e1f27f2%7C0%7C0%7C639254038688014321%7CUnknown%7CTWFpbGZsb3d8eyJFbXB0eU1hcGkiOnRydWUsIlYiOiIwLjAuMDAwMCIsIlAiOiJXaW4zMiIsIkFOIjoiTWFpbCIsIldUIjoyfQ%3D%3D%7C0%7C%7C%7C&sdata=S4qPFkVdn2y1F%2FtLRWhlkqwazsOUbNVrPEi%2FYuHoN7w%3D&reserved=0 > > )<https://github.com/apache/airflow/pull/62922>, Amogh raised a naming > > > concern about the Dynamic Task Batching API that I would like to settle > > > here before the PR merges, > > > since renaming afterwards would be a breaking change. > > > > > > ## What `.batch(size=N)` does today > > > > > > `task.batch(size=17).iterate(url=urls)` creates 17 mapped task > instances > > > through Dynamic Task > > > Mapping, and each of them iterates over its share of `urls` in a single > > > task instance using > > > Iterable Tasks. The items are distributed round robin: item `i` goes to > > > task instance `i % 17`. > > > So `size` is the number of task instances, not a chunk length, and the > > > task instances do not > > > receive contiguous slices of the input. > > > > > > The round robin scheme is deliberate. The number of task instances has > to > > > be fixed before the > > > underlying iterable is consumed, because the scheduler needs the > mapping > > > cardinality up front. > > > With round robin that number is simply `size`. Contiguous chunking, the > > > way `itertools.batched` > > > works, would need `ceil(len(items) / size)` task instances, which is > > > unknowable until a possibly > > > unbounded or paginated iterable has been fully drained. That would > defeat > > > the purpose of iterating > > > lazily. > > > > > > ## The concern > > > > > > The name reads like `itertools.batched(iterable, size)`, where `size` > is > > > the chunk length and the > > > number of chunks is derived from it. Ours is the opposite: the number > of > > > task instances is given, > > > and the per-instance share is derived. The class docstring and the Task > > > SDK docs currently spend a > > > full paragraph undoing that first impression. A name that carries the > > > meaning directly would not > > > need it. > > > > > > ## Options > > > > > > 1. Keep `.batch(size=N)` and rely on documentation. Lowest churn, but > the > > > mismatch with > > > `itertools.batched` stays and every new reader has to be corrected. > > > > > > 2. Keep the method, rename the parameter: `.batch(count=N)` or > > > `.batch(tasks=N)`. Keeps the > > > "Dynamic Task Batching" vocabulary from the AIP, and `count` or > > `tasks` > > > says what the number > > > is. `.batch(tasks=17).iterate(url=urls)` reads as "spread over 17 > > > tasks, then iterate". > > > > > > 3. Rename the method as well, to something that describes partitioning > > > rather than chunking, for > > > example `.partition(count=N)`, `.shard(count=N)` or > > > `.spread(across=N)`. Most descriptive, but > > > drifts from the AIP's own terminology, and "shard" and "partition" > > > carry data-engineering > > > connotations that may suggest contiguous ranges just as much as > > "batch" > > > does. > > > > > > My preference is option 2 with `count=`. It keeps the API shape and the > > > AIP terminology, fixes > > > the misleading part, and costs nothing now because the feature is > > > unreleased. `size` would be > > > kept as a rejected alias raising a clear error rather than silently > > > accepted, so nobody copies > > > the old spelling from an early draft. > > > > > > Unless there are objections or a better name comes up, I will treat > this > > > as lazy consensus in > > > 72 hours and update the PR, the docs and the AIP page accordingly. > > > > > > Thanks, > > > David > > > > > > General (Internal Property) > > > > > >
