[issue34014] loop.run_in_executor should propagate current contextvars

Andrew Svetlov added the comment: contextvars and run_in_executor() cannot be used together with process executor. asyncio.to_thread() runs with a copy of the current context. Available since Python 3.9 https://docs.python.org/3/library/asyncio-task.html?highlight=to_thread#asyncio.to_thread

[issue34014] loop.run_in_executor should propagate current contextvars

Change by Cheryl Sabella : -- versions: +Python 3.9 -Python 3.8 ___ Python tracker ___ ___ Python-bugs-list mailing list

[issue34014] loop.run_in_executor should propagate current contextvars

Viktor Kovtun added the comment: Hey, as I see there is no new API yet. Can we take a look again on 3) ? I'll be happy to update PR 9688 -- ___ Python tracker ___

[issue34014] loop.run_in_executor should propagate current contextvars

Change by Yury Selivanov : -- status: pending -> open versions: -Python 3.7 ___ Python tracker ___ ___ Python-bugs-list mailing

[issue34014] loop.run_in_executor should propagate current contextvars

Yury Selivanov added the comment: > It is a new feature for Python 3.8 anyway, no need to rush Yep, I agree. Let's see if we end up having a new nice high-level API in 3.8; if not we go for (3). -- resolution: -> postponed status: open -> pending type: -> enhancement

[issue34014] loop.run_in_executor should propagate current contextvars

Andrew Svetlov added the comment: I prefer (4) but can live with (3) It is a new feature for Python 3.8 anyway, no need to rush -- ___ Python tracker ___

[issue34014] loop.run_in_executor should propagate current contextvars

Yury Selivanov added the comment: [Andrew] > I vote on not changing `run_in_executor` behavior if we cannot make it work > with `ProcessPoolExecutor`. > If a new API will solve the problem -- that's fine. Until it landed the explicit context propagation is the satisfactory solution. I'm

[issue34014] loop.run_in_executor should propagate current contextvars

Viktor Kovtun added the comment: I've created new pull request https://github.com/python/cpython/pull/9688 with the implementation of proposed changes -- ___ Python tracker

[issue34014] loop.run_in_executor should propagate current contextvars

Change by Viktor Kovtun : -- pull_requests: +9074 ___ Python tracker ___ ___ Python-bugs-list mailing list Unsubscribe:

[issue34014] loop.run_in_executor should propagate current contextvars

Andrew Svetlov added the comment: I vote on not changing `run_in_executor` behavior if we cannot make it work with `ProcessPoolExecutor`. If a new API will solve the problem -- that's fine. Until it landed the explicit context propagation is the satisfactory solution. --

[issue34014] loop.run_in_executor should propagate current contextvars

Yury Selivanov added the comment: One problem with (3) is what will happen if someone uses "retain_context=True" and a ProcessPoolExecutor. It has to fail in a graceful and obvious way. -- ___ Python tracker

[issue34014] loop.run_in_executor should propagate current contextvars

Yury Selivanov added the comment: So we're deprecating passing non-ThreadPoolExecutor instances to loop.set_default_executor. In 3.9 that will trigger an error. For this issue we have basically the next few options: (1) Do nothing; (2) Fix "run_in_executor" to start copying and running in

[issue34014] loop.run_in_executor should propagate current contextvars

Viktor Kovtun added the comment: `ProcessPoolExecutor as executor is not so popular in real world` - as executor for asyncio -- ___ Python tracker ___

[issue34014] loop.run_in_executor should propagate current contextvars

Viktor Kovtun added the comment: What about to check instance of executor and depending on that propagate contextvars or not? As well to raise RuntimeError for ProcessPoolExecutor with provided context? I assume, that ProcessPoolExecutor as executor is not so popular in real world, but I

[issue34014] loop.run_in_executor should propagate current contextvars

Yury Selivanov added the comment: As a workaround you can subclass the ThreadPoolExecutor and set it as a default executor. -- ___ Python tracker ___

[issue34014] loop.run_in_executor should propagate current contextvars

Yury Selivanov added the comment: I considered enabling that, but in the end decided not to. The reason is that it's possible to use a ProcessPoolExecutor with run_in_execuror(), and context cars currently don't support pickling (and probably never will). We can't have a single api that

[issue34014] loop.run_in_executor should propagate current contextvars

Change by Viktor Kovtun : -- keywords: +patch pull_requests: +7645 stage: -> patch review ___ Python tracker ___ ___

[issue34014] loop.run_in_executor should propagate current contextvars

New submission from Viktor Kovtun : PEP 567 supports copying context into another threads, but for asyncio users each call `run_in_executor` requires explicit context propagation For end users expected behavior that context is copied automatically -- components: asyncio messages: