Sharding is usually (or at least every time I’ve seen it used) when work is distributed based on some hash function, batching is a simpler thing that is closer to the current implementation - take the first n items, run them, get the next n items, run them etc. shard I would expect to be somewhere halfway between batched and full task mapping, i.e. send the items to 3 workers/tasks in parallel.
`shard(3)` to me could read as "shard into 3 tasks at a time” — or at least I’d have to check the first time I come across it. This could be fixable by not allowing positional args, only named ones. I’m also now questioning the “Dynamic” part of it. What about this is dynamic? In Dynamic Task Mapping, things to change at runtime (i.e the scheduler dynamically creates more Tis as execution happens), but that isn’t what is happening here is it? From the AIP alone (I haven’t had time to read the PR) — which now I see it is a secondary issue David: the AIP is defined in terms of the PR. That is not the purpose of an AIP - it should be a (mostly) stand-alone proposal of a new feature. Or to ask it another way: Why does it need its own name? Could it be as simple as adding a batching feature on to Mapped Tasks? “This is a set of dynamic mapped tasks” “this is a set of batched mapped tasks” etc? (If everything is “Dynamic this” or “Dynamic that” then it starts to lose meaning.) Sorry, this turned out as more of a stream of consciousness than I intended. -ash > On 4 Jun 2026, at 21:05, Natanel <[email protected]> wrote: > > Hello, > > I think that it would be better to go with shard or chunk, personally I > would prefer shard as it seems to work like sharding (at least according to > what David showed during the devcall) as data was not split to slices, > rather was sharded between the two dynamically created tasks > Batch just seem a little confusing as if I see a method with task.batch(3) > I would not know if it meant splitting to batches of 3 or to 3 batches, > same as with chunk though less confusing, if I see some code with > task.shard(3) I don't think anyone would confuse it with shards of size 3, > at least that is my opinion, let me know what you think > > Thanks, > Natanel. > > > On Thu, Jun 4, 2026, 21:53 Sameer Mesiah <[email protected]> wrote: > >> Hi David, >> >> I would go with either Batch or Chunk. >> >> But I must say I have no strong opinions with regards to the naming of this >> feature. >> >> Thanks, >> Sameer Mesiah. >> >> On Thu, 4 Jun 2026 at 17:56, Blain David <[email protected]> wrote: >> >>> Hi all, >>> >>> We need a better name than partition for Dynamic Task Partitioning. >>> >>> The main issue is that partition already strongly suggests asset/data >>> partitions in Airflow, >>> so using the same word here creates avoidable confusion for users and >>> contributors. >>> >>> We’d like a term that is clear, intuitive, and doesn’t overlap with >>> existing Airflow concepts. >>> >>> Some alternatives raised so far during the devcall: >>> >>> >>> * >>> batch (e.g. Dynamic Task Batching) >>> * >>> chunk (e.g. Dynamic Task Chunking) >>> * >>> slice (bit confusing but chose to still mention it anway) >>> * >>> shard >>> * >>> segment >>> >>> >>> My current lean is towards chunk and batch. It feels familiar, readable >> in >>> both code and docs, and avoids the existing partition/data-partition >>> association. >>> >>> I’d love feedback on: >>> >>> >>> * >>> which term feels most natural >>> * >>> which term is least ambiguous >>> * >>> or whether there’s a better option we haven’t considered? >>> >>> >>> One note: map was mentioned as well, but that seems too close to existing >>> task.map() terminology. >>> >>> Please share thoughts, especially if you have concerns about any of the >>> options above or a stronger suggestion for the long-term name. >>> >>> Naming is indeed hard 🙂 >>> >>> Kind regards, >>> David >>> >> --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
