On 14Jun2018 1214, Chris Barker via Python-Dev wrote:
Excuse my ignorance (or maybe it's a vocabulary thing), but I'm trying
to understand the problem here.
But if I have this right:
I've been using asyncio a lot lately and have encountered this
problem several times. Imagine you want to do a lot of queries
against a database, spawning 10000 tasks in parallel will probably
cause a lot of them to fail.
async is not parallel -- all the tasks will be run in the same thread
(Unless you explicitly spawn another thread), and only one task is
running at once, and the task switching happens when the task
specifically releases itself.
If the task isn't actually doing the work, but merely waiting for it to
finish, then you can end up overloading the thing that *is* doing the
task (e.g. the network interface, database server, other thread/process,
file system, etc.).
Single-threaded async is actually all about *waiting* - it provides a
convenient model to do other tasks while you are waiting for the first
(as well as a convenient model to indicate what should be done after it
completes - there are two conveniences here).
If the underlying thing you're doing *can* run in parallel, but becomes
less efficient the more times you do it (for example, most file system
operations fall into this category), you will want to limit how many
tasks you *start*, not just how many you are waiting for. I often use
semaphores for this when I need it, and it looks like
asyncio.Semaphore() is sufficient for this:
import asyncio
task_limiter = asyncio.Semaphore(4)
async def my_task():
await task_limiter.acquire()
try:
await do_db_request()
finally:
task_limiter.release()
Cheers,
Steve
_______________________________________________
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe:
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
