+1
On 05.10.26 11:45, Shahar Epstein wrote:
+1 (binding), thanks for leading this effort!
Shahar
On Thu, Oct 1, 2026, 11:00 Blain David <[email protected]> wrote:
Hi everyone,
I would like to restart the vote on AIP-104, now titled Iterable Tasks
(IT) and Task Spreading (TS):
https://cwiki.apache.org/confluence/spaces/AIRFLOW/pages/421954527/AIP-104+Iterable+Tasks+IT+and+Task+Spreading+TS
The previous vote, under the name "Task Iteration and Dynamic Task
Batching", ran during the summer and did not get much engagement:
https://lists.apache.org/thread/x5bd26zfqkkhp7sx7nb0crjlsctf2z5j
The AIP has changed since then, so please vote on the current page rather
than on what you may remember:
- Renamed "Task Iteration (TI)" to "Iterable Tasks (IT)", as TI is already
widely used for Task Instance (as remarked by Daniel).
- Renamed "Dynamic Task Batching (DTB)" to "Task Spreading (TS)". The
primitive is now .spread(across=N) instead of .batch(size=N), as "size"
read like a chunk length while it is the number of Task Instances. Items
are distributed round robin over those N Task Instances. The new name was
discussed and voted on in separate threads (linked below).
- Iterable Tasks now checkpoint progress per item through AIP-103 Task
State. On a retry, or after a worker crash, items that already finished are
not run again; only the items that failed or were in flight are.
- The page now documents task_concurrency, the setting that controls how
many items run at once inside one Task Instance.
- The page now lists the current limits: deferrable operators,
reschedule-mode sensors and TriggerDagRunOperator are not supported inside
.iterate() yet (deferrable support is planned in a separate PR);
execution_timeout applies per item only to async operators;
multiple_outputs is ignored; and pools count the Task Instance rather than
its items.
The previous discussion threads can be found here:
https://lists.apache.org/thread/ztnfsqolow4v1zsv4pkpnxc1fk0hbf2p
https://lists.apache.org/thread/tr3x4sx55k0fk2coq28xs2648m1p65cc
The naming of the spread primitive:
https://lists.apache.org/thread/6qlbvvnt76514rstzk447pvs4784wcj2
(discussion)
https://lists.apache.org/thread/rrc9bmhh5dcpyj4rcpjr2lj1q8vjdoxp (vote)
Thank you again to everyone who reviewed the AIP and took part in the
discussions. A special thank you to Kaxil for the many detailed reviews,
and to TP, Jens, Jarek and Ash for their comments. Thanks as well to
Daniel, Jedidiah and Amogh for their more recent reviews on the AIP page,
and to everyone who voted on the naming of the spread primitive.
The vote will remain open for 5 days and will close on Wednesday, 7
October 2026, at 00:00 GMT.
Everyone is encouraged to vote, although only votes from PMC members and
committers are considered binding.
Please vote accordingly:
[ ] +1 Approve
[ ] +0 no opinion
[ ] -1 disapprove with the reason
Please consider this my +1 (binding).
Regards,
David
General (Internal Property)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
