**Sorry, did not know markdown is supported**
At work we faced a problem of long running python code. Our case was a short
task, but huge count of iterations. Something like:
``` python
for x in data_list:
# do 1ms non-io pure python task
```
So we block loop for more than 100ms, or even 1000ms. The first naive solution
was "move this to thread" so python will switch thread context and asyncio loop
will not be blocked, but this raised two problems:
* If we do asyncio things in our task (create Future in our case), then we need
to pass loop explicitly and use call_soon_threadsafe
* We still saw asyncio warnings about blocking the loop. Not sure why, but
maybe because of GIL was not released when main/asyncio thread became active.
We endend up with wrapper for iterable, which switch «asyncio context» via
asyncio.sleep(0) (since we know that sleep(0) have special code, which just
switches context) by time or by count. Here is our code:
``` python
async def iterate_non_blocking(iterable, context_switch_interval=0.01,
context_switch_count=None):
last_context_switch_time = time.perf_counter()
for i, value in enumerate(iterable, start=1):
yield value
switch_context_by_interval = context_switch_interval and \
(time.perf_counter() - last_context_switch_time) >=
context_switch_interval
switch_context_by_count = context_switch_count and i %
context_switch_count == 0
if switch_context_by_interval or switch_context_by_count:
await asyncio.sleep(0)
last_context_switch_time = time.perf_counter()
```
I'm not sure if this is a good approach, but this solves all our problems for
this kind of blocking cases. Here comes discussable things:
* Is there a better approach for such cases?
* Should this be a part of asyncio?
* Should this be a part of documentation as recipe?
_______________________________________________
Python-ideas mailing list -- python-ideas@python.org
To unsubscribe send an email to python-ideas-le...@python.org
https://mail.python.org/mailman3/lists/python-ideas.python.org/
Message archived at
https://mail.python.org/archives/list/python-ideas@python.org/message/P5P3KC36QT6TE6VRALLR5I7JMLW7F3ET/
Code of Conduct: http://python.org/psf/codeofconduct/
