At work we faced a problem of long running python code. Our case was a short task, but huge count of iterations. Something like:
for x in data_list: # do 1ms non-io pure python task So we block loop for more than 100ms, or even 1000ms. The first naive solution was "move this to thread" so python will switch thread context and asyncio loop will not be blocked, but this raised two problems: * If we do asyncio things in our task (create `Future` in our case), then we need to pass loop explicitly and use `call_soon_threadsafe` * We still saw asyncio warnings about blocking the loop. Not sure why, but maybe because of GIL was not released when main/asyncio thread became active. We endend up with wrapper for iterable, which switch «asyncio context» via `asyncio.sleep(0)` (since we know that sleep(0) have special code, which just switches context) by time or by count. Here is our code: async def iterate_non_blocking(iterable, context_switch_interval=0.01, context_switch_count=None): last_context_switch_time = time.perf_counter() for i, value in enumerate(iterable, start=1): yield value switch_context_by_interval = context_switch_interval and \ (time.perf_counter() - last_context_switch_time) >= context_switch_interval switch_context_by_count = context_switch_count and i % context_switch_count == 0 if switch_context_by_interval or switch_context_by_count: await asyncio.sleep(0) last_context_switch_time = time.perf_counter() I'm not sure if this is a good approach, but this solves all our problems for this kind of blocking cases. Here comes discussable things: * Is there a better approach for such cases? * Should this be a part of asyncio? * Should this be a part of documentation as recipe? _______________________________________________ Python-ideas mailing list -- python-ideas@python.org To unsubscribe send an email to python-ideas-le...@python.org https://mail.python.org/mailman3/lists/python-ideas.python.org/ Message archived at https://mail.python.org/archives/list/python-ideas@python.org/message/EZWJS6OTBDG7LCX2WECDMPET6I5BMGDU/ Code of Conduct: http://python.org/psf/codeofconduct/