[issue33115] Asyncio loop blocks with a lot of parallel tasks

[Terry J. Reedy]

Terry J. Reedy  added the comment:

If the unforeseeable future arrives, someone can reopen or open a new issue.

--
nosy: +terry.reedy
resolution:  -> wont fix
stage:  -> resolved
status: open -> closed

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue33115] Asyncio loop blocks with a lot of parallel tasks

[Marat Sharafutdinov]

Marat Sharafutdinov  added the comment:

Concerning the example adding a jitter is useful, thanks!

But anyway in case there will be something not constant as sleeping for 1 sec 
is, the problem will continue to appear.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue33115] Asyncio loop blocks with a lot of parallel tasks

[Andrew Svetlov]

Andrew Svetlov  added the comment:

The problem of the example is: all 1 tasks starts in the same moment, than 
waits for 1 sec each and at the same moment every task clones itself.

Adding a jitter into example can solve the issue.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue33115] Asyncio loop blocks with a lot of parallel tasks

[Yury Selivanov]

Yury Selivanov  added the comment:

> Does this mean that GC uses most part of CPU time so the loop blocks?

GC stops all Python code in the OS process from running.  Because of the GIL 
code in threads will obviously be stopped too.  This is true for both CPython 
and PyPy at this moment.

> And another question: do you have any plans to optimize the loop so it would 
> be possible to run really lot of tasks in parallel?

The only way of doing this is to have a few asyncio OS processes (because of 
the GIL we can't implement M:N scheduling in a single Python process).  So it's 
not going to happen in the foreseeable future :(

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue33115] Asyncio loop blocks with a lot of parallel tasks

[Marat Sharafutdinov]

Marat Sharafutdinov  added the comment:

Does this mean that GC uses most part of CPU time so the loop blocks?

And another question: do you have any plans to optimize the loop so it would be 
possible to run really lot of tasks in parallel?

Thanks.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue33115] Asyncio loop blocks with a lot of parallel tasks

[Yury Selivanov]

Yury Selivanov  added the comment:

The "blocking" you observe is caused by Python GC.  If I add "import gc; 
gc.disable()" the warnings disappear.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue33115] Asyncio loop blocks with a lot of parallel tasks

[Marat Sharafutdinov]

Marat Sharafutdinov  added the comment:

But why if I use multiprocessing (run 100 tasks by 100 workers) it still 
continue blocking loop within some workers? Are 100 tasks "a lot of work" for 
asyncio loop?

```python
import asyncio
from multiprocessing import Process

worker_count = 100
task_count = 100

def worker_main(worker_id):
async def main():
for x in range(1, task_count + 1):
asyncio.ensure_future(f(x))

async def f(x):
if x % 1000 == 0 or x == task_count:
print(f'[WORKER-{worker_id}] Run f({x})')
await asyncio.sleep(1)
loop.call_later(1, lambda: asyncio.ensure_future(f(x)))

loop = asyncio.get_event_loop()
loop.set_debug(True)
loop.run_until_complete(main())
loop.run_forever()

if __name__ == '__main__':
for worker_id in range(worker_count):
worker = Process(target=worker_main, args=(worker_id,), daemon=True)
worker.start()
while True:
pass
```

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue33115] Asyncio loop blocks with a lot of parallel tasks

[Yury Selivanov]

Yury Selivanov  added the comment:

Well, there's nothing we can do here, it's just a lot of work for a 
single-threaded process to get a 1 tasks going.  You'll get the same 
picture in any other async Python framework.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue33115] Asyncio loop blocks with a lot of parallel tasks

[Marat Sharafutdinov]

New submission from Marat Sharafutdinov :

I want to schedule a lot of parallel tasks, but it becomes slow with loop 
blocking:

```python
import asyncio

task_count = 1

async def main():
for x in range(1, task_count + 1):
asyncio.ensure_future(f(x))

async def f(x):
if x % 1000 == 0 or x == task_count:
print(f'Run f({x})')
await asyncio.sleep(1)
loop.call_later(1, lambda: asyncio.ensure_future(f(x)))

loop = asyncio.get_event_loop()
loop.set_debug(True)
loop.run_until_complete(main())
loop.run_forever()
```

Outputs:
```
Executing  
result=None created at /usr/lib/python3.6/asyncio/base_events.py:446> took 
0.939 seconds

...

Executing , None) at 
/usr/lib/python3.6/asyncio/futures.py:339 created at 
/usr/lib/python3.6/asyncio/tasks.py:480> took 0.113 seconds

...

Executing  wait_for=()] created at 
/usr/lib/python3.6/asyncio/base_events.py:275> created at test_aio.py:13> took 
0.100 seconds

...
```

What can be another way to schedule a lot of parallel tasks?

--
components: asyncio
messages: 314207
nosy: asvetlov, decaz, yselivanov
priority: normal
severity: normal
status: open
title: Asyncio loop blocks with a lot of parallel tasks
type: resource usage
versions: Python 3.6

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com