And I can get behind `.spread(across_n_tasks=5)` sort of thing (I think that is slightly clearer than just .spread(across=5)
-ash > On 24 Sep 2026, at 11:24, Ash Berlin-Taylor <[email protected]> wrote: > > Oh, I possibly posted in the wrong thread. Repeating here > > Sooo I was throwing some ideas around with David in a DM, and here are the > ones I came up with > > some_task.batch(num_tasks=5)? > some_task.batch(across=5) ? > some_task.spread(across=5) ? > some_task.parallelise(across=5)? > some_task.distribute(across=5)? > some_task.distribute(across_n_tasks=5)? > > (I also thought about something like `group` or `group_by`, but couldn’t work > out something that wasn’t > > I think I like the `.parallelise(across=n)` the most? It’s not perfect I > admit. > > Also (perhaps oddly) I think even with that I think the feature should be > called "Batched Mapped Task” even if the fn is .parallelise or .distribute — > but I don’t think "Task Spreading” is very clear from a user-reading-the-docs > PoV. > > So I think I’m +1 to the verb being batch, even if I think the method should > be “parallelise”. (I’m definitely inhabiting the Lawful Chaotic character > alignment grid square with this vote!) > > -1 to partition, as I think that needs to be reserved for Asset Partitions > (i.e. partitions in data, not in processing) > > -a > >> On 24 Sep 2026, at 08:13, Shahar Epstein <[email protected]> wrote: >> >> +1 (binding) >> >> spread > scatter > unfold > partition > shard > batch >> >> >> Shahar >> >> On Thu, Sep 24, 2026 at 10:01 AM Blain David <[email protected]> >> wrote: >> >>> Hi all, >>> >>> The discussion thread [1] about the name of the method that distributes an >>> iterable over a fixed number of mapped task instances brought up several >>> candidates without converging on one, so I am opening a vote to settle it >>> before PR #62922 [2] merges. >>> What the method does >>> `task.<name>(N).iterate(url=urls)` creates N mapped task instances through >>> Dynamic Task Mapping, and each instance iterates over its share of `urls` >>> in a single task instance using Iterable Tasks. Items are distributed round >>> robin: item `i` goes to task instance `i % N`. N is the number of task >>> instances, not a chunk length. That is why `batch(size=N)` was found >>> misleading: it reads like `itertools.batched`, which does the opposite >>> (chunk length given, number of chunks derived). >>> Candidates >>> In the order they came up in the discussion, with the spelling each would >>> take: >>> >>> 1. >>> batch - `.batch(tasks=N)` "Task Batching" (original name in the AIP) >>> 2. >>> spread - `.spread(across=N)` "Task Spreading" (Shahar, Constance; >>> currently on the AIP page) >>> 3. >>> scatter - `.scatter(across=N)` "Task Scattering" (Jarek) >>> 4. >>> unfold - `.unfold(into=N)` "Task Unfolding" (Jarek) >>> 5. >>> shard - `.shard(into=N)` "Task Sharding" >>> 6. >>> partition - `.partition(into=N)` "Task Partitioning" >>> >>> The keyword spellings are indicative. The vote is about the verb; the >>> exact keyword is settled in the PR once the verb is chosen. >>> Arguments raised so far >>> >>> * >>> `batch` and `shard` are semantically overloaded (Shahar). Naming it after >>> * >>> `itertools.batched` is confusing because the semantics are the opposite >>> (Ash). >>> * >>> `shard` and `partition` carry data-engineering connotations that suggest >>> contiguous >>> * >>> ranges as much as `batch` does. Partitions are now an Airflow feature, so >>> `partition` >>> would introduce a name clash (Constance). >>> * >>> `spread` is hard to misread because of the preposition: "spread across 17" >>> cannot be >>> * >>> read as "17 each" (Constance). >>> * >>> Jarek pointed out that the stdlib settled on a positional `n` for >>> `batched(it, n)`, >>> * >>> which is a precedent to match or consciously break if `batch` is kept. >>> >>> Regardless of the outcome >>> >>> * >>> The parameter is keyword-only. `spread(17)` is as ambiguous as `batch(17)` >>> and is rejected. >>> * >>> Passing `size=` raises a clear error instead of being accepted silently. >>> * >>> The docs explain the round-robin distribution and the reason for it. >>> * >>> The AIP-104 page [3], the Task SDK docs and PR #62922 are updated to the >>> winning name >>> * >>> before the PR merges. >>> >>> How to vote >>> Reply to this thread with your preferences ranked from most to least >>> preferred, for example: >>> >>> spread > scatter > batch >>> >>> You do not have to rank all six. The result is decided by instant runoff: >>> the option with the fewest first preferences is eliminated and its votes >>> move to the next preference on each ballot, until one option holds a >>> majority. >>> >>> Everyone is welcome to vote. Votes from committers and PMC members are >>> binding. >>> The vote stays open for at least 72 hours and closes on Monday 28 >>> September 2026 at 12:00 CEST. I will post the result in this thread. >>> >>> [1] https://lists.apache.org/thread/6qlbvvnt76514rstzk447pvs4784wcj2 >>> [2] https://github.com/apache/airflow/pull/62922 >>> [3] >>> https://cwiki.apache.org/confluence/spaces/AIRFLOW/pages/421954527/AIP-104+Iterable+Tasks+IT+and+Task+Spreading+TS >>> >>> Thanks, >>> David >>> >>> >>> General (Internal Property) >>> > > > --------------------------------------------------------------------- > To unsubscribe, e-mail: [email protected] > For additional commands, e-mail: [email protected] > --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
