+1 (binding) for spread(across=N) On 2026/09/24 14:18:16 Jarek Potiuk wrote: > +1 (binding) on spread(across=N) > > On Thu, Sep 24, 2026 at 12:35 PM David Blain <[email protected]> wrote: > > > > > > > On 2026/09/24 07:01:24 Blain David wrote: > > > Hi all, > > > > > > The discussion thread [1] about the name of the method that distributes > > an iterable over a fixed number of mapped task instances brought up several > > candidates without converging on one, so I am opening a vote to settle it > > before PR #62922 [2] merges. > > > What the method does > > > `task.<name>(N).iterate(url=urls)` creates N mapped task instances > > through Dynamic Task Mapping, and each instance iterates over its share of > > `urls` in a single task instance using Iterable Tasks. Items are > > distributed round robin: item `i` goes to task instance `i % N`. N is the > > number of task instances, not a chunk length. That is why `batch(size=N)` > > was found misleading: it reads like `itertools.batched`, which does the > > opposite (chunk length given, number of chunks derived). > > > Candidates > > > In the order they came up in the discussion, with the spelling each > > would take: > > > > > > 1. batch - `.batch(tasks=N)` "Task Batching" (original name in the AIP) > > > 2. spread - `.spread(across=N)` "Task Spreading" (Shahar, Constance; > > currently on the AIP page) > > > 3. scatter - `.scatter(across=N)` "Task Scattering" (Jarek) > > > 4. unfold - `.unfold(into=N)` "Task Unfolding" (Jarek) > > > 5. shard - `.shard(into=N)` "Task Sharding" > > > 6. partition - `.partition(into=N)` "Task Partitioning" > > > > > > The keyword spellings are indicative. The vote is about the verb; the > > exact keyword is settled in the PR once the verb is chosen. > > > Arguments raised so far > > > > > > * `batch` and `shard` are semantically overloaded (Shahar). Naming it > > after > > > * `itertools.batched` is confusing because the semantics are the > > opposite (Ash). > > > * `shard` and `partition` carry data-engineering connotations that > > suggest contiguous > > > * ranges as much as `batch` does. Partitions are now an Airflow > > feature, so `partition` > > > would introduce a name clash (Constance). > > > * `spread` is hard to misread because of the preposition: "spread > > across 17" cannot be > > > * read as "17 each" (Constance). > > > * Jarek pointed out that the stdlib settled on a positional `n` for > > `batched(it, n)`, which is a precedent to match or consciously break if > > `batch` is kept. > > > > > > Regardless of the outcome > > > > > > * The parameter is keyword-only. `spread(17)` is as ambiguous as > > `batch(17)` and is rejected. > > > * Passing `size=` raises a clear error instead of being accepted > > silently. > > > * The docs explain the round-robin distribution and the reason for it. > > > * The AIP-104 page [3], the Task SDK docs and PR #62922 are updated to > > the winning name before the PR merges. > > > > > > How to vote > > > Reply to this thread with your preferences ranked from most to least > > preferred, for example: > > > > > > spread > scatter > batch > > > > > > You do not have to rank all six. The result is decided by instant > > runoff: the option with the fewest first preferences is eliminated and its > > votes move to the next preference on each ballot, until one option holds a > > majority. > > > > > > Everyone is welcome to vote. Votes from committers and PMC members are > > binding. > > > The vote stays open for at least 72 hours and closes on Monday 28 > > September 2026 at 12:00 CEST. I will post the result in this thread. > > > > > > [1] https://lists.apache.org/thread/6qlbvvnt76514rstzk447pvs4784wcj2 > > > [2] https://github.com/apache/airflow/pull/62922 > > > [3] > > https://cwiki.apache.org/confluence/spaces/AIRFLOW/pages/421954527/AIP-104+Iterable+Tasks+IT+and+Task+Spreading+TS > > > > > > Thanks, > > > David > > > > > > > > > General (Internal Property) > > > > > > > --------------------------------------------------------------------- > > To unsubscribe, e-mail: [email protected] > > For additional commands, e-mail: [email protected] > > > > >
--------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
