Oh, I possibly posted in the wrong thread. Repeating here Sooo I was throwing some ideas around with David in a DM, and here are the ones I came up with
some_task.batch(num_tasks=5)? some_task.batch(across=5) ? some_task.spread(across=5) ? some_task.parallelise(across=5)? some_task.distribute(across=5)? some_task.distribute(across_n_tasks=5)? (I also thought about something like `group` or `group_by`, but couldn’t work out something that wasn’t I think I like the `.parallelise(across=n)` the most? It’s not perfect I admit. Also (perhaps oddly) I think even with that I think the feature should be called "Batched Mapped Task” even if the fn is .parallelise or .distribute — but I don’t think "Task Spreading” is very clear from a user-reading-the-docs PoV. So I think I’m +1 to the verb being batch, even if I think the method should be “parallelise”. (I’m definitely inhabiting the Lawful Chaotic character alignment grid square with this vote!) -1 to partition, as I think that needs to be reserved for Asset Partitions (i.e. partitions in data, not in processing) -a > On 24 Sep 2026, at 08:13, Shahar Epstein <[email protected]> wrote: > > +1 (binding) > > spread > scatter > unfold > partition > shard > batch > > > Shahar > > On Thu, Sep 24, 2026 at 10:01 AM Blain David <[email protected]> > wrote: > >> Hi all, >> >> The discussion thread [1] about the name of the method that distributes an >> iterable over a fixed number of mapped task instances brought up several >> candidates without converging on one, so I am opening a vote to settle it >> before PR #62922 [2] merges. >> What the method does >> `task.<name>(N).iterate(url=urls)` creates N mapped task instances through >> Dynamic Task Mapping, and each instance iterates over its share of `urls` >> in a single task instance using Iterable Tasks. Items are distributed round >> robin: item `i` goes to task instance `i % N`. N is the number of task >> instances, not a chunk length. That is why `batch(size=N)` was found >> misleading: it reads like `itertools.batched`, which does the opposite >> (chunk length given, number of chunks derived). >> Candidates >> In the order they came up in the discussion, with the spelling each would >> take: >> >> 1. >> batch - `.batch(tasks=N)` "Task Batching" (original name in the AIP) >> 2. >> spread - `.spread(across=N)` "Task Spreading" (Shahar, Constance; >> currently on the AIP page) >> 3. >> scatter - `.scatter(across=N)` "Task Scattering" (Jarek) >> 4. >> unfold - `.unfold(into=N)` "Task Unfolding" (Jarek) >> 5. >> shard - `.shard(into=N)` "Task Sharding" >> 6. >> partition - `.partition(into=N)` "Task Partitioning" >> >> The keyword spellings are indicative. The vote is about the verb; the >> exact keyword is settled in the PR once the verb is chosen. >> Arguments raised so far >> >> * >> `batch` and `shard` are semantically overloaded (Shahar). Naming it after >> * >> `itertools.batched` is confusing because the semantics are the opposite >> (Ash). >> * >> `shard` and `partition` carry data-engineering connotations that suggest >> contiguous >> * >> ranges as much as `batch` does. Partitions are now an Airflow feature, so >> `partition` >> would introduce a name clash (Constance). >> * >> `spread` is hard to misread because of the preposition: "spread across 17" >> cannot be >> * >> read as "17 each" (Constance). >> * >> Jarek pointed out that the stdlib settled on a positional `n` for >> `batched(it, n)`, >> * >> which is a precedent to match or consciously break if `batch` is kept. >> >> Regardless of the outcome >> >> * >> The parameter is keyword-only. `spread(17)` is as ambiguous as `batch(17)` >> and is rejected. >> * >> Passing `size=` raises a clear error instead of being accepted silently. >> * >> The docs explain the round-robin distribution and the reason for it. >> * >> The AIP-104 page [3], the Task SDK docs and PR #62922 are updated to the >> winning name >> * >> before the PR merges. >> >> How to vote >> Reply to this thread with your preferences ranked from most to least >> preferred, for example: >> >> spread > scatter > batch >> >> You do not have to rank all six. The result is decided by instant runoff: >> the option with the fewest first preferences is eliminated and its votes >> move to the next preference on each ballot, until one option holds a >> majority. >> >> Everyone is welcome to vote. Votes from committers and PMC members are >> binding. >> The vote stays open for at least 72 hours and closes on Monday 28 >> September 2026 at 12:00 CEST. I will post the result in this thread. >> >> [1] https://lists.apache.org/thread/6qlbvvnt76514rstzk447pvs4784wcj2 >> [2] https://github.com/apache/airflow/pull/62922 >> [3] >> https://cwiki.apache.org/confluence/spaces/AIRFLOW/pages/421954527/AIP-104+Iterable+Tasks+IT+and+Task+Spreading+TS >> >> Thanks, >> David >> >> >> General (Internal Property) >> --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
