[issue30782] Allow limiting the number of concurrent tasks in asyncio.as_completed

Change by Jack Wong : -- nosy: +iforapsy ___ Python tracker ___ ___ Python-bugs-list mailing list Unsubscribe:

[issue30782] Allow limiting the number of concurrent tasks in asyncio.as_completed

Yury Selivanov added the comment: > it will help simplify some Mock things. Yeah, we'll need to chat about that so I can use Mock requirements in the PEP. :) -- ___ Python tracker

[issue30782] Allow limiting the number of concurrent tasks in asyncio.as_completed

Lisa Roach added the comment: Oh nice, I remember talking to you about the MultiError before, it will help simplify some Mock things. Happy to help out if you want more eyes on it. On Mon, Sep 9, 2019 at 3:46 PM Yury Selivanov wrote: > > Yury Selivanov added the comment: > > FWIW I've been

[issue30782] Allow limiting the number of concurrent tasks in asyncio.as_completed

Yury Selivanov added the comment: FWIW I've been using TaskGroups in EdgeDB codebase extensively: https://github.com/edgedb/edgedb/blob/master/edb/common/taskgroup.py (you can use the code, it's Apache 2) The only thing that prevented us from merging them in 3.8 is that we need to formally

[issue30782] Allow limiting the number of concurrent tasks in asyncio.as_completed

Lisa Roach added the comment: Sounds good, thanks for the explanation Yury. I look forward to the TaskGroups! -- ___ Python tracker ___

[issue30782] Allow limiting the number of concurrent tasks in asyncio.as_completed

Yury Selivanov added the comment: We want to add TaskGroups to asyncio (a similar concept to Trio's nurseries). TaskGroups use the `async with` statement to clearly define where Tasks are created and at which point they are expected to be completed or destroyed. asyncio.gather(),

[issue30782] Allow limiting the number of concurrent tasks in asyncio.as_completed

Lisa Roach added the comment: I would like to see this implemented. I run into memory and speed issues when running with 1000+ tasks quite frequently, and have to write my own rate limiters around it. It doesn't look to me that it is adding a large amount of complexity to as_completed.

[issue30782] Allow limiting the number of concurrent tasks in asyncio.as_completed

Andrey Paramonov added the comment: Hello! Below is updated implementation containing more consistent error handling. The main rough edges encountered: 1. asyncio.Queue alone proved insufficient for precise control of limit, as asyncio.create_task() schedules created Task() immediately and

[issue30782] Allow limiting the number of concurrent tasks in asyncio.as_completed

Andrey Paramonov added the comment: > an implicit requirement that back pressure from the consumer should be > handled (i.e. if whoever's iterating through "async for fut in > as_completed(...)" is too slow, then the tasks should pause until it catches > up) No, I don't think it is

[issue30782] Allow limiting the number of concurrent tasks in asyncio.as_completed

twisteroid ambassador added the comment: I feel like once you lay out all the requirements: taking futures from an (async) generator, limiting the number of concurrent tasks, getting completed tasks to one consumer "as completed", and an implicit requirement that back pressure from the

[issue30782] Allow limiting the number of concurrent tasks in asyncio.as_completed

Andrey Paramonov added the comment: Might as_completed() be considered a low-level API, but as of Python 3.7 there are seemingly no ready alternatives to achieve proposed behavior. All of asyncio.gather(), asyncio.wait(), asyncio.as_completed() expect awaitables list of limited size; doing

[issue30782] Allow limiting the number of concurrent tasks in asyncio.as_completed

glin added the comment: @Andrew Svetlov: I was surprised when you wrote "as_completed() is low-level API.", so I wondered what is high level API and I googled it and first that came was official doc: https://docs.python.org/3/library/asyncio-api-index.html Where it's written that it's high

[issue30782] Allow limiting the number of concurrent tasks in asyncio.as_completed

Andrew Svetlov added the comment: Third parties are not python core devs responsibility. I don't aware about existing library with such functionality. -- ___ Python tracker

[issue30782] Allow limiting the number of concurrent tasks in asyncio.as_completed

Andy Balaam added the comment: I would argue that this makes as_completed a lot more useful than it is now, so would be worth adding (maybe after 3.7). But, if this does not go into asyncio, is there another library where it would belong? Or should this be a

[issue30782] Allow limiting the number of concurrent tasks in asyncio.as_completed

Yury Selivanov added the comment: I agree, let's keep as_completed() simple for now. Handling generators+async correctly is hard, so we definitely don't have time for this in 3.7. -- versions: +Python 3.8 -Python 3.7 ___

[issue30782] Allow limiting the number of concurrent tasks in asyncio.as_completed

Andrew Svetlov added the comment: as_completed() is low-level API. Let's not overload it with different parameters. Anyway `as_completed()` uses only asyncio.Future and it's public API like `add_done_callback()` etc. You can master everything what you need without

[issue30782] Allow limiting the number of concurrent tasks in asyncio.as_completed

Andy Balaam added the comment: bump -- ___ Python tracker ___ ___ Python-bugs-list mailing list Unsubscribe:

[issue30782] Allow limiting the number of concurrent tasks in asyncio.as_completed

Changes by Andy Balaam : -- pull_requests: +2475 ___ Python tracker ___ ___

[issue30782] Allow limiting the number of concurrent tasks in asyncio.as_completed

New submission from Andy Balaam: asyncio.as_completed allows us to provide lots of coroutines (or Futures) to schedule, and then deal with the results as soon as they are available, in a loop, or a streaming style. I propose to allow as_completed to work on very large numbers of coroutines,