Yeah. After the discussion, spread is a clear winner for me. On Thu, Sep 24, 2026 at 3:13 PM constance.astronomer.io via dev < [email protected]> wrote:
> +1 to keeping the feature name as is. > > For the function name, my vote is still some_task.spread(across=N), but > between parallelize and distribute it’s just a matter of personal > preference. It makes the behaviour really obvious, sounds more natural and > is more readable (to me) and doesn’t imply that all mapped tasks will > literally run in parallel at the exact same time. > > -1 to _(across_n_tasks=N). It’s unnecessary with the verbs in the running. > > > On Sep 24, 2026, at 6:20 AM, Ash Berlin-Taylor <[email protected]> wrote: > > > > Sooo I was throwing some ideas around with David in a DM, and here are > the ones I came up with > > > > some_task.batch(num_tasks=5)? > > some_task.batch(across=5) ? > > some_task.spread(across=5) ? > > some_task.parallelise(across=5)? > > some_task.distribute(across=5)? > > some_task.distribute(across_n_tasks=5)? > > > > I think I like the `.parallelise(across=n)` the most? It’s not perfect I > admit. > > > > Also (perhaps oddly) I think even with that I think the feature should > be called "Batched Mapped Task” even if the fn is .parallelise or > .distribute — but I don’t think "Task Spreading” is very clear from a > user-reading-the-docs PoV. > > > > -a > > > >> On 24 Sep 2026, at 09:51, Amogh Desai <[email protected]> wrote: > >> > >> Late to the party, but thanks, David. > >> > >> Out of all the options, I vote +1 for `.spread(across=N)`. > >> > >> -1 to unfold because, to me, it means generating a sequence from > something, > >> which describes > >> creating items rather than distributing them. > >> > >> And regardless of the name, the round-robin distribution needs to be > >> explicit in the SDK docs with the reasoning, > >> as Constance said. The name fixes the "how many" confusion, not the > "which > >> items go where" one. > >> > >> Thanks & Regards, > >> Amogh Desai > >> > >> > >>> On Thu, Sep 24, 2026 at 1:48 AM Jarek Potiuk <[email protected]> wrote: > >>> > >>> How about "scatter" or "unfold" :)? > >>> > >>> (just had a little conversation with Claude about it). > >>> > >>> On Wed, Sep 23, 2026 at 12:20 PM Blain David <[email protected]> > >>> wrote: > >>> > >>>> I've updated the AIP-104 confluence page and choose the spread option > and > >>>> thus the Task Spreading instead of Batched Task Mapping. > >>>> > >>>> Does everyone agree with this change? > >>>> > >>>> > >>>> > >>> > https://cwiki.apache.org/confluence/spaces/AIRFLOW/pages/421954527/AIP-104+Iterable+Tasks+IT+and+Task+Spreading+TS > >>>> > >>>> If so, then I will push those changes in the PR as well: > >>>> > >>>> https://github.com/apache/airflow/pull/62922 > >>>> > >>>> General (Internal Property) > >>>> ________________________________ > >>>> From: Shahar Epstein <[email protected]> > >>>> Sent: Saturday, September 19, 2026 10:37 > >>>> To: [email protected] <[email protected]> > >>>> Subject: Re: [DISCUSS] Naming of `.batch(size=)` in AIP-104 (Iterable > >>>> Tasks and Dynamic Task Batching) > >>>> > >>>> EXTERNAL MAIL: Indien je de afzender van deze e-mail niet kent en deze > >>>> niet vertrouwt, klik niet op een link of open geen bijlages. Bij > twijfel, > >>>> stuur deze e-mail als bijlage naar [email protected]<mailto: > >>>> [email protected]>. > >>>> > >>>> I'd go with .spread(...), as both batch(...) and shard(...) are > already > >>>> semantically overloaded. > >>>> > >>>> > >>>> Shahar > >>>> > >>>> On Thu, Sep 17, 2026 at 12:26 PM Blain David <[email protected] > > > >>>> wrote: > >>>> > >>>>> Hi all, > >>>>> > >>>>> While reviewing the AIP-104 PR ( > >>>>> > >>>> > >>> > https://eur01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fapache%2Fairflow%2Fpull%2F62922&data=05%7C02%7Cdavid.blain%40infrabel.be%7Cfb61e6c745c34a77080408df1629466b%7Cb82bc314ab8e4d6fb18946f02e1f27f2%7C0%7C0%7C639254038688014321%7CUnknown%7CTWFpbGZsb3d8eyJFbXB0eU1hcGkiOnRydWUsIlYiOiIwLjAuMDAwMCIsIlAiOiJXaW4zMiIsIkFOIjoiTWFpbCIsIldUIjoyfQ%3D%3D%7C0%7C%7C%7C&sdata=S4qPFkVdn2y1F%2FtLRWhlkqwazsOUbNVrPEi%2FYuHoN7w%3D&reserved=0 > >>>> )<https://github.com/apache/airflow/pull/62922>, Amogh raised a > naming > >>>>> concern about the Dynamic Task Batching API that I would like to > settle > >>>>> here before the PR merges, > >>>>> since renaming afterwards would be a breaking change. > >>>>> > >>>>> ## What `.batch(size=N)` does today > >>>>> > >>>>> `task.batch(size=17).iterate(url=urls)` creates 17 mapped task > >>> instances > >>>>> through Dynamic Task > >>>>> Mapping, and each of them iterates over its share of `urls` in a > single > >>>>> task instance using > >>>>> Iterable Tasks. The items are distributed round robin: item `i` goes > to > >>>>> task instance `i % 17`. > >>>>> So `size` is the number of task instances, not a chunk length, and > the > >>>>> task instances do not > >>>>> receive contiguous slices of the input. > >>>>> > >>>>> The round robin scheme is deliberate. The number of task instances > has > >>> to > >>>>> be fixed before the > >>>>> underlying iterable is consumed, because the scheduler needs the > >>> mapping > >>>>> cardinality up front. > >>>>> With round robin that number is simply `size`. Contiguous chunking, > the > >>>>> way `itertools.batched` > >>>>> works, would need `ceil(len(items) / size)` task instances, which is > >>>>> unknowable until a possibly > >>>>> unbounded or paginated iterable has been fully drained. That would > >>> defeat > >>>>> the purpose of iterating > >>>>> lazily. > >>>>> > >>>>> ## The concern > >>>>> > >>>>> The name reads like `itertools.batched(iterable, size)`, where `size` > >>> is > >>>>> the chunk length and the > >>>>> number of chunks is derived from it. Ours is the opposite: the number > >>> of > >>>>> task instances is given, > >>>>> and the per-instance share is derived. The class docstring and the > Task > >>>>> SDK docs currently spend a > >>>>> full paragraph undoing that first impression. A name that carries the > >>>>> meaning directly would not > >>>>> need it. > >>>>> > >>>>> ## Options > >>>>> > >>>>> 1. Keep `.batch(size=N)` and rely on documentation. Lowest churn, but > >>> the > >>>>> mismatch with > >>>>> `itertools.batched` stays and every new reader has to be corrected. > >>>>> > >>>>> 2. Keep the method, rename the parameter: `.batch(count=N)` or > >>>>> `.batch(tasks=N)`. Keeps the > >>>>> "Dynamic Task Batching" vocabulary from the AIP, and `count` or > >>>> `tasks` > >>>>> says what the number > >>>>> is. `.batch(tasks=17).iterate(url=urls)` reads as "spread over 17 > >>>>> tasks, then iterate". > >>>>> > >>>>> 3. Rename the method as well, to something that describes > partitioning > >>>>> rather than chunking, for > >>>>> example `.partition(count=N)`, `.shard(count=N)` or > >>>>> `.spread(across=N)`. Most descriptive, but > >>>>> drifts from the AIP's own terminology, and "shard" and "partition" > >>>>> carry data-engineering > >>>>> connotations that may suggest contiguous ranges just as much as > >>>> "batch" > >>>>> does. > >>>>> > >>>>> My preference is option 2 with `count=`. It keeps the API shape and > the > >>>>> AIP terminology, fixes > >>>>> the misleading part, and costs nothing now because the feature is > >>>>> unreleased. `size` would be > >>>>> kept as a rejected alias raising a clear error rather than silently > >>>>> accepted, so nobody copies > >>>>> the old spelling from an early draft. > >>>>> > >>>>> Unless there are objections or a better name comes up, I will treat > >>> this > >>>>> as lazy consensus in > >>>>> 72 hours and update the PR, the docs and the AIP page accordingly. > >>>>> > >>>>> Thanks, > >>>>> David > >>>>> > >>>>> General (Internal Property) > >>>>> > >>>> > >>> > > > > > > --------------------------------------------------------------------- > > To unsubscribe, e-mail: [email protected] > > For additional commands, e-mail: [email protected] > > > > --------------------------------------------------------------------- > To unsubscribe, e-mail: [email protected] > For additional commands, e-mail: [email protected] > >
