[issue34014] loop.run_in_executor should propagate current contextvars

[Andrew Svetlov]

Andrew Svetlov  added the comment:

contextvars and run_in_executor() cannot be used together with process executor.

asyncio.to_thread() runs with a copy of the current context.
Available since Python 3.9

https://docs.python.org/3/library/asyncio-task.html?highlight=to_thread#asyncio.to_thread

--
resolution: postponed -> out of date
stage: patch review -> resolved
status: open -> closed

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue34014] loop.run_in_executor should propagate current contextvars

[Cheryl Sabella]

Change by Cheryl Sabella :


--
versions: +Python 3.9 -Python 3.8

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue34014] loop.run_in_executor should propagate current contextvars

[Viktor Kovtun]

Viktor Kovtun  added the comment:

Hey, as I see there is no new API yet.

Can we take a look again on 3) ?

I'll be happy to update PR 9688

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue34014] loop.run_in_executor should propagate current contextvars

[Yury Selivanov]

Change by Yury Selivanov :


--
status: pending -> open
versions:  -Python 3.7

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue34014] loop.run_in_executor should propagate current contextvars

[Yury Selivanov]

Yury Selivanov  added the comment:

> It is a new feature for Python 3.8 anyway, no need to rush

Yep, I agree. Let's see if we end up having a new nice high-level API in 3.8; 
if not we go for (3).

--
resolution:  -> postponed
status: open -> pending
type:  -> enhancement

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue34014] loop.run_in_executor should propagate current contextvars

[Andrew Svetlov]

Andrew Svetlov  added the comment:

I prefer (4) but can live with (3)

It is a new feature for Python 3.8 anyway, no need to rush

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue34014] loop.run_in_executor should propagate current contextvars

[Yury Selivanov]

Yury Selivanov  added the comment:

[Andrew]
> I vote on not changing `run_in_executor` behavior if we cannot make it work 
> with `ProcessPoolExecutor`.

> If a new API will solve the problem -- that's fine.
Until it landed the explicit context propagation is the satisfactory solution.


I'm not sure I understand.  Should we close this issue or you're OK with (3)?

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue34014] loop.run_in_executor should propagate current contextvars

[Viktor Kovtun]

Viktor Kovtun  added the comment:

I've created new pull request https://github.com/python/cpython/pull/9688 with 
the implementation of proposed changes

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue34014] loop.run_in_executor should propagate current contextvars

[Viktor Kovtun]

Change by Viktor Kovtun :


--
pull_requests: +9074

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue34014] loop.run_in_executor should propagate current contextvars

[Andrew Svetlov]

Andrew Svetlov  added the comment:

I vote on not changing `run_in_executor` behavior if we cannot make it work 
with `ProcessPoolExecutor`.

If a new API will solve the problem -- that's fine.
Until it landed the explicit context propagation is the satisfactory solution.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue34014] loop.run_in_executor should propagate current contextvars

[Yury Selivanov]

Yury Selivanov  added the comment:

One problem with (3) is what will happen if someone uses "retain_context=True" 
and a ProcessPoolExecutor.  It has to fail in a graceful and obvious way.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue34014] loop.run_in_executor should propagate current contextvars

[Yury Selivanov]

Yury Selivanov  added the comment:

So we're deprecating passing non-ThreadPoolExecutor instances to 
loop.set_default_executor.  In 3.9 that will trigger an error.

For this issue we have basically the next few options:

(1) Do nothing;

(2) Fix "run_in_executor" to start copying and running in correct context 
automatically

(3) Add a new keyword-only parameter to "run_in_executor": 
"retain_context=False"

(4) Design a new async/await friendly API for using thread- and process-pools.

Now, (4) will happen.  The "run_in_executor" method is low-level and requires 
accessing the event loop to use it.  We probably have enough time to design 
this new API before 3.8.

As for implementing (3) in 3.8 -- I'd be OK with that too.  Andrew, your 
thoughts?

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue34014] loop.run_in_executor should propagate current contextvars

[Viktor Kovtun]

Viktor Kovtun  added the comment:

`ProcessPoolExecutor as executor is not so popular in real world` - as executor 
for asyncio

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue34014] loop.run_in_executor should propagate current contextvars

[Viktor Kovtun]

Viktor Kovtun  added the comment:

What about to check instance of executor and depending on that propagate 
contextvars or not?

As well to raise RuntimeError for ProcessPoolExecutor with provided context?

I assume, that ProcessPoolExecutor as executor is not so popular in real world, 
but I can not provide any stats.

Different behavior is defiantly at least weird, but do not support natively 
what is technically expected also strange.

https://docs.python.org/3.7/library/contextvars.html#asyncio-support `Context 
variables are natively supported in asyncio` is hard for end users.

Each use case of contextvars + run_in_executor will produce copy-paste 
boilerplate

As for me it is very similar to the `asyncio.get_event_loop()` history

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue34014] loop.run_in_executor should propagate current contextvars

[Yury Selivanov]

Yury Selivanov  added the comment:

As a workaround you can subclass the ThreadPoolExecutor and set it as a default 
executor.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue34014] loop.run_in_executor should propagate current contextvars

[Yury Selivanov]

Yury Selivanov  added the comment:

I considered enabling that, but in the end decided not to. The reason is that 
it's possible to use a ProcessPoolExecutor with run_in_execuror(), and context 
cars currently don't support pickling (and probably never will). We can't have 
a single api that sometimes works with contextvars and sometimes doesn't.

So this is a "won't fix".

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue34014] loop.run_in_executor should propagate current contextvars

[Viktor Kovtun]

Change by Viktor Kovtun :


--
keywords: +patch
pull_requests: +7645
stage:  -> patch review

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue34014] loop.run_in_executor should propagate current contextvars

[Viktor Kovtun]

New submission from Viktor Kovtun :

PEP 567 supports copying context into another threads, but for asyncio users 
each call `run_in_executor` requires explicit context propagation

For end users expected behavior that context is copied automatically

--
components: asyncio
messages: 320814
nosy: asvetlov, hellysmile, yselivanov
priority: normal
severity: normal
status: open
title: loop.run_in_executor should propagate current contextvars
versions: Python 3.7, Python 3.8

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com