Hi all,
The discussion thread [1] about the name of the method that distributes an
iterable over a fixed number of mapped task instances brought up several
candidates without converging on one, so I am opening a vote to settle it
before PR #62922 [2] merges.
What the method does
`task.<name>(N).iterate(url=urls)` creates N mapped task instances through
Dynamic Task Mapping, and each instance iterates over its share of `urls` in a
single task instance using Iterable Tasks. Items are distributed round robin:
item `i` goes to task instance `i % N`. N is the number of task instances, not
a chunk length. That is why `batch(size=N)` was found misleading: it reads like
`itertools.batched`, which does the opposite (chunk length given, number of
chunks derived).
Candidates
In the order they came up in the discussion, with the spelling each would take:
1.
batch - `.batch(tasks=N)` "Task Batching" (original name in the AIP)
2.
spread - `.spread(across=N)` "Task Spreading" (Shahar, Constance; currently on
the AIP page)
3.
scatter - `.scatter(across=N)` "Task Scattering" (Jarek)
4.
unfold - `.unfold(into=N)` "Task Unfolding" (Jarek)
5.
shard - `.shard(into=N)` "Task Sharding"
6.
partition - `.partition(into=N)` "Task Partitioning"
The keyword spellings are indicative. The vote is about the verb; the exact
keyword is settled in the PR once the verb is chosen.
Arguments raised so far
*
`batch` and `shard` are semantically overloaded (Shahar). Naming it after
*
`itertools.batched` is confusing because the semantics are the opposite (Ash).
*
`shard` and `partition` carry data-engineering connotations that suggest
contiguous
*
ranges as much as `batch` does. Partitions are now an Airflow feature, so
`partition`
would introduce a name clash (Constance).
*
`spread` is hard to misread because of the preposition: "spread across 17"
cannot be
*
read as "17 each" (Constance).
*
Jarek pointed out that the stdlib settled on a positional `n` for `batched(it,
n)`,
*
which is a precedent to match or consciously break if `batch` is kept.
Regardless of the outcome
*
The parameter is keyword-only. `spread(17)` is as ambiguous as `batch(17)` and
is rejected.
*
Passing `size=` raises a clear error instead of being accepted silently.
*
The docs explain the round-robin distribution and the reason for it.
*
The AIP-104 page [3], the Task SDK docs and PR #62922 are updated to the
winning name
*
before the PR merges.
How to vote
Reply to this thread with your preferences ranked from most to least preferred,
for example:
spread > scatter > batch
You do not have to rank all six. The result is decided by instant runoff: the
option with the fewest first preferences is eliminated and its votes move to
the next preference on each ballot, until one option holds a majority.
Everyone is welcome to vote. Votes from committers and PMC members are binding.
The vote stays open for at least 72 hours and closes on Monday 28 September
2026 at 12:00 CEST. I will post the result in this thread.
[1] https://lists.apache.org/thread/6qlbvvnt76514rstzk447pvs4784wcj2
[2] https://github.com/apache/airflow/pull/62922
[3]
https://cwiki.apache.org/confluence/spaces/AIRFLOW/pages/421954527/AIP-104+Iterable+Tasks+IT+and+Task+Spreading+TS
Thanks,
David
General (Internal Property)
