Re: [Python-Dev] PEP 567 v3

[Koos Zevenhoven]

On Thu, Jan 18, 2018 at 3:53 AM, Yury Selivanov 
wrote:

​[]


>
> Given the time frame of the Python 3.7 release schedule it was decided
> to defer this proposal to Python 3.8.
>

​It occurs to me that I had misread this to refer to the whole PEP.
Although I thought it's kind of sad that after all this, contextvars still
would not make it into 3.7, I also thought that it might be the right
decision. As you may already know, I think there are several problems with
this PEP. Would it be worth it to write down some thoughts on this PEP in
the morning?

-- Koos​



-- 
+ Koos Zevenhoven + http://twitter.com/k7hoven +
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 567 v3

[Nathaniel Smith]

On Thu, Jan 18, 2018 at 12:03 AM, Antoine Pitrou  wrote:
> On Wed, 17 Jan 2018 20:53:42 -0500
> Yury Selivanov  wrote:
>>
>> Proposed by Antoine Pitrou, this could enable transparent
>> cross-process use of ``Context`` objects, so the
>> `Offloading execution to other threads`_ example would work with
>> a ``ProcessPoolExecutor`` too.
>>
>> Enabling this is problematic because of the following reasons:
>>
>> 1. ``ContextVar`` objects do not have ``__module__`` and
>>``__qualname__`` attributes, making straightforward pickling
>>of ``Context`` objects impossible.  This is solvable by modifying
>>the API to either auto detect the module where a context variable
>>is defined, or by adding a new keyword-only "module" parameter
>>to ``ContextVar`` constructor.
>>
>> 2. Not all context variables refer to picklable objects.  Making a
>>``ContextVar`` picklable must be an opt-in.
>
> This is a red herring.  If a value isn't picklable, pickle will simply
> raise as it does in other contexts.  You should't need to opt in for
> anything here.

The complication is that Contexts collect ContextVars from all over
the process. So if people are going to pickle Contexts, we need some
mechanism to make sure that we don't end up in a situation where it
seems to work and users depend on it, and then they import a new
library and suddenly pickling raises an error (because the new library
internally uses a ContextVar that happens not to be pickleable).

-n

-- 
Nathaniel J. Smith -- https://vorpus.org
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 567 v3

[Antoine Pitrou]

On Wed, 17 Jan 2018 20:53:42 -0500
Yury Selivanov  wrote:
> 
> Proposed by Antoine Pitrou, this could enable transparent
> cross-process use of ``Context`` objects, so the
> `Offloading execution to other threads`_ example would work with
> a ``ProcessPoolExecutor`` too.
> 
> Enabling this is problematic because of the following reasons:
> 
> 1. ``ContextVar`` objects do not have ``__module__`` and
>``__qualname__`` attributes, making straightforward pickling
>of ``Context`` objects impossible.  This is solvable by modifying
>the API to either auto detect the module where a context variable
>is defined, or by adding a new keyword-only "module" parameter
>to ``ContextVar`` constructor.
> 
> 2. Not all context variables refer to picklable objects.  Making a
>``ContextVar`` picklable must be an opt-in.

This is a red herring.  If a value isn't picklable, pickle will simply
raise as it does in other contexts.  You should't need to opt in for
anything here.

Regards

Antoine.
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 567 v3

[Yury Selivanov]

On Wed, Jan 17, 2018 at 8:53 PM, Yury Selivanov  wrote:
> On Wed, Jan 17, 2018 at 2:24 PM, Guido van Rossum  
> wrote:
>> Perhaps you can update the PEP with a summary of the rejected ideas from
>> this thread?
>
> The Rejected Ideas section of the PEP is now updated with the below:

I've added two more subsections to Rejected Ideas:


Make Context a MutableMapping
-

Making the ``Context`` class implement the ``abc.MutableMapping``
interface would mean that it is possible to set and unset variables
using ``Context[var] = value`` and ``del Context[var]`` operations.

This proposal was deferred to Python 3.8+ because of the following:

1. If in Python 3.8 it is decided that generators should support
   context variables (see :pep:`550` and :pep:`568`), then ``Context``
   would be transformed into a chain-map of context variables mappings
   (as every generator would have its own mapping).  That would make
   mutation operations like ``Context.__delitem__`` confusing, as
   they would operate only on the topmost mapping of the chain.

2. Having a single way of mutating the context
   (``ContextVar.set()`` and ``ContextVar.reset()`` methods) makes
   the API more straightforward.

   For example, it would be non-obvious why the below code fragment
   does not work as expected::

 var = ContextVar('var')

 ctx = copy_context()
 ctx[var] = 'value'
 print(ctx[var])  # Prints 'value'

 print(var.get())  # Raises a LookupError

   While the following code would work::

 ctx = copy_context()

 def func():
 ctx[var] = 'value'

 # Contrary to the previous example, this would work
 # because 'func()' is running within 'ctx'.
 print(ctx[var])
 print(var.get())

 ctx.run(func)


Have initial values for ContextVars
---

Nathaniel Smith proposed to have a required ``initial_value``
keyword-only argument for the ``ContextVar`` constructor.

The main argument against this proposal is that for some types
there is simply no sensible "initial value" except ``None``.
E.g. consider a web framework that stores the current HTTP
request object in a context variable.  With the current semantics
it is possible to create a context variable without a default value::

# Framework:
current_request: ContextVar[Request] = \
ContextVar('current_request')


# Later, while handling an HTTP request:
request: Request = current_request.get()

# Work with the 'request' object:
return request.method

Note that in the above example there is no need to check if
``request`` is ``None``.  It is simply expected that the framework
always sets the ``current_request`` variable, or it is a bug (in
which case ``current_request.get()`` would raise a ``LookupError``).

If, however, we had a required initial value, we would have
to guard against ``None`` values explicitly::

# Framework:
current_request: ContextVar[Optional[Request]] = \
ContextVar('current_request', initial_value=None)


# Later, while handling an HTTP request:
request: Optional[Request] = current_request.get()

# Check if the current request object was set:
if request is None:
raise RuntimeError

# Work with the 'request' object:
return request.method

Moreover, we can loosely compare context variables to regular
Python variables and to ``threading.local()`` objects.  Both
of them raise errors on failed lookups (``NameError`` and
``AttributeError`` respectively).

Yury
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 567 v3

[Yury Selivanov]

On Wed, Jan 17, 2018 at 2:24 PM, Guido van Rossum  wrote:
> Perhaps you can update the PEP with a summary of the rejected ideas from
> this thread?

The Rejected Ideas section of the PEP is now updated with the below:

Token.reset() instead of ContextVar.reset()
---

Nathaniel Smith suggested to implement the ``ContextVar.reset()``
method directly on the ``Token`` class, so instead of::

token = var.set(value)
# ...
var.reset(token)

we would write::

token = var.set(value)
# ...
token.reset()

Having ``Token.reset()`` would make it impossible for a user to
attempt to reset a variable with a token object created by another
variable.

This proposal was rejected for the reason of ``ContextVar.reset()``
being clearer to the human reader of the code which variable is
being reset.


Make Context objects picklable
--

Proposed by Antoine Pitrou, this could enable transparent
cross-process use of ``Context`` objects, so the
`Offloading execution to other threads`_ example would work with
a ``ProcessPoolExecutor`` too.

Enabling this is problematic because of the following reasons:

1. ``ContextVar`` objects do not have ``__module__`` and
   ``__qualname__`` attributes, making straightforward pickling
   of ``Context`` objects impossible.  This is solvable by modifying
   the API to either auto detect the module where a context variable
   is defined, or by adding a new keyword-only "module" parameter
   to ``ContextVar`` constructor.

2. Not all context variables refer to picklable objects.  Making a
   ``ContextVar`` picklable must be an opt-in.

Given the time frame of the Python 3.7 release schedule it was decided
to defer this proposal to Python 3.8.


Yury
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 567 v3

[Victor Stinner]

FYI In the PEP 540, I didn't try to elaborate on each design change, but I
wrote a very short version history at the end:
https://www.python.org/dev/peps/pep-0540/#version-history

Maybe something like that would help for the PEP 567?

Victor

Le 17 janv. 2018 8:26 PM, "Guido van Rossum"  a
écrit :

> Perhaps you can update the PEP with a summary of the rejected ideas from
> this thread?
>
> On Jan 17, 2018 7:23 AM, "Yury Selivanov"  wrote:
>
>> On Wed, Jan 17, 2018 at 6:03 AM, Antoine Pitrou 
>> wrote:
>> > On Tue, 16 Jan 2018 17:18:06 -0800
>> > Nathaniel Smith  wrote:
>> >> On Tue, Jan 16, 2018 at 5:06 PM, Yury Selivanov <
>> yselivanov...@gmail.com> wrote:
>> >> >
>> >> > I think it would be a very fragile thing In practice: if you have
>> even
>> >> > one variable in the context that isn't pickleable, your code that
>> uses
>> >> > a ProcessPool would stop working.  I would defer Context
>> pickleability
>> >> > to 3.8+.
>> >>
>> >> There's also a more fundamental problem: you need some way to match up
>> >> the ContextVar objects across the two processes, and right now they
>> >> don't have an attached __module__ or __qualname__.
>> >
>> > They have a name, though.  So perhaps the name could serve as a unique
>> > identifier?  Instead of being serialized as a bunch of ContextVars, the
>> > Context would then be serialized as a {name: value} dict.
>>
>> One of the points of the ContextVar design is to avoid having unique
>> identifiers requirement. Names can clash which leads to data being
>> lost. If you prohibit them from clashing, then if libraries A and B
>> happen to use the same context variable name—you can't use them both
>> in your projects.  And without enforcing name uniqueness, your
>> approach to serialize context as a dict with string keys won't work.
>>
>> I like Nathaniel's idea to explicitly enable ContextVars pickling
>> support on a per-var basis.  Unfortunately we don't have time to
>> seriously consider and debate (and implement!) this idea in time
>> before the 3.7 freeze.
>>
>> In the meanwhile, given that Context objects are fully introspectable,
>> users can implement their own ad-hoc solutions for serializers or
>> cross-process execution.
>>
>> Yury
>> ___
>> Python-Dev mailing list
>> Python-Dev@python.org
>> https://mail.python.org/mailman/listinfo/python-dev
>> Unsubscribe: https://mail.python.org/mailman/options/python-dev/guido%
>> 40python.org
>>
>
> ___
> Python-Dev mailing list
> Python-Dev@python.org
> https://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe: https://mail.python.org/mailman/options/python-dev/
> victor.stinner%40gmail.com
>
>
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 567 v3

[Guido van Rossum]

Perhaps you can update the PEP with a summary of the rejected ideas from
this thread?

On Jan 17, 2018 7:23 AM, "Yury Selivanov"  wrote:

> On Wed, Jan 17, 2018 at 6:03 AM, Antoine Pitrou 
> wrote:
> > On Tue, 16 Jan 2018 17:18:06 -0800
> > Nathaniel Smith  wrote:
> >> On Tue, Jan 16, 2018 at 5:06 PM, Yury Selivanov <
> yselivanov...@gmail.com> wrote:
> >> >
> >> > I think it would be a very fragile thing In practice: if you have even
> >> > one variable in the context that isn't pickleable, your code that uses
> >> > a ProcessPool would stop working.  I would defer Context pickleability
> >> > to 3.8+.
> >>
> >> There's also a more fundamental problem: you need some way to match up
> >> the ContextVar objects across the two processes, and right now they
> >> don't have an attached __module__ or __qualname__.
> >
> > They have a name, though.  So perhaps the name could serve as a unique
> > identifier?  Instead of being serialized as a bunch of ContextVars, the
> > Context would then be serialized as a {name: value} dict.
>
> One of the points of the ContextVar design is to avoid having unique
> identifiers requirement. Names can clash which leads to data being
> lost. If you prohibit them from clashing, then if libraries A and B
> happen to use the same context variable name—you can't use them both
> in your projects.  And without enforcing name uniqueness, your
> approach to serialize context as a dict with string keys won't work.
>
> I like Nathaniel's idea to explicitly enable ContextVars pickling
> support on a per-var basis.  Unfortunately we don't have time to
> seriously consider and debate (and implement!) this idea in time
> before the 3.7 freeze.
>
> In the meanwhile, given that Context objects are fully introspectable,
> users can implement their own ad-hoc solutions for serializers or
> cross-process execution.
>
> Yury
> ___
> Python-Dev mailing list
> Python-Dev@python.org
> https://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe: https://mail.python.org/mailman/options/python-dev/
> guido%40python.org
>
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 567 v3

[Yury Selivanov]

On Wed, Jan 17, 2018 at 6:03 AM, Antoine Pitrou  wrote:
> On Tue, 16 Jan 2018 17:18:06 -0800
> Nathaniel Smith  wrote:
>> On Tue, Jan 16, 2018 at 5:06 PM, Yury Selivanov  
>> wrote:
>> >
>> > I think it would be a very fragile thing In practice: if you have even
>> > one variable in the context that isn't pickleable, your code that uses
>> > a ProcessPool would stop working.  I would defer Context pickleability
>> > to 3.8+.
>>
>> There's also a more fundamental problem: you need some way to match up
>> the ContextVar objects across the two processes, and right now they
>> don't have an attached __module__ or __qualname__.
>
> They have a name, though.  So perhaps the name could serve as a unique
> identifier?  Instead of being serialized as a bunch of ContextVars, the
> Context would then be serialized as a {name: value} dict.

One of the points of the ContextVar design is to avoid having unique
identifiers requirement. Names can clash which leads to data being
lost. If you prohibit them from clashing, then if libraries A and B
happen to use the same context variable name—you can't use them both
in your projects.  And without enforcing name uniqueness, your
approach to serialize context as a dict with string keys won't work.

I like Nathaniel's idea to explicitly enable ContextVars pickling
support on a per-var basis.  Unfortunately we don't have time to
seriously consider and debate (and implement!) this idea in time
before the 3.7 freeze.

In the meanwhile, given that Context objects are fully introspectable,
users can implement their own ad-hoc solutions for serializers or
cross-process execution.

Yury
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 567 v3

[Antoine Pitrou]

On Tue, 16 Jan 2018 17:18:06 -0800
Nathaniel Smith  wrote:
> On Tue, Jan 16, 2018 at 5:06 PM, Yury Selivanov  
> wrote:
> >
> > I think it would be a very fragile thing In practice: if you have even
> > one variable in the context that isn't pickleable, your code that uses
> > a ProcessPool would stop working.  I would defer Context pickleability
> > to 3.8+.  
> 
> There's also a more fundamental problem: you need some way to match up
> the ContextVar objects across the two processes, and right now they
> don't have an attached __module__ or __qualname__.

They have a name, though.  So perhaps the name could serve as a unique
identifier?  Instead of being serialized as a bunch of ContextVars, the
Context would then be serialized as a {name: value} dict.

Regards

Antoine.
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 567 v3

[Nick Coghlan]

On 17 January 2018 at 11:27, Nathaniel Smith  wrote:
> On Tue, Jan 16, 2018 at 2:44 PM, Yury Selivanov  
> wrote:
>> 4. ContextVar.reset(token) now raises a ValueError if the token was
>> created in a different Context.
>
> A minor bit of polish: given that Token objects have to track the
> associated ContextVar anyway, I think it'd be cleaner if instead of
> doing:
>
> token = cvar.set(...)
> cvar.reset(token)
>
> we made the API be:
>
> token = cvar.set(...)
> token.reset()

As a counterpoint to this, consider the case where you're working with
*two* cvars:


token1 = cvar1.set(...)
token2 = cvar2.set(...)
...
cvar1.reset(token1)
...
cvar2.reset(token2)

At the point where the resets happen, you know exactly which cvar is
being reset, even if you don't know where the token was created.

With reset-on-the-token, you're entirely reliant on variable naming to
know which ContextVar is going to be affected:

token1 = cvar1.set(...)
token2 = cvar2.set(...)
...
token1.reset() # Resets cvar1
...
token2.reset() # Resets cvar2

If someone really does want an auto-reset API, it's also fairly easy
to build atop the more explicit one:

def set_cvar(cvar, value):
token = cvar.set(value)
return functools.partial(cvar.reset, token)

reset_cvar1 = set_cvar(cvar1, ...)
...
reset_cvar1()

Cheers,
Nick.

-- 
Nick Coghlan   |   ncogh...@gmail.com   |   Brisbane, Australia
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 567 v3

[Guido van Rossum]

On Tue, Jan 16, 2018 at 5:33 PM, Yury Selivanov 
wrote:

> On Tue, Jan 16, 2018 at 8:27 PM, Nathaniel Smith  wrote:
> [..]
> > token = cvar.set(...)
> > token.reset()
>
> I see the point, but I think that having the 'reset' method defined on
> the ContextVar class is easier to grasp.  It also feels natural that a
> pair of set/reset methods is defined on the same class.  This is
> highly subjective though, so let's see which option Guido likes more.
>

I think this came up in one of the previous reviews of the PEP. I like
Yury's (redundant) version -- it makes it clear to the human reader of the
code which variable is being reset. And it's not like it's going to be used
that much -- it'll be likely hidden inside a context manager.

-- 
--Guido van Rossum (python.org/~guido)
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 567 v3

[Yury Selivanov]

On Tue, Jan 16, 2018 at 8:27 PM, Nathaniel Smith  wrote:
[..]
> token = cvar.set(...)
> token.reset()

I see the point, but I think that having the 'reset' method defined on
the ContextVar class is easier to grasp.  It also feels natural that a
pair of set/reset methods is defined on the same class.  This is
highly subjective though, so let's see which option Guido likes more.

Yury
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 567 v3

[Nathaniel Smith]

On Tue, Jan 16, 2018 at 2:44 PM, Yury Selivanov  wrote:
> 4. ContextVar.reset(token) now raises a ValueError if the token was
> created in a different Context.

A minor bit of polish: given that Token objects have to track the
associated ContextVar anyway, I think it'd be cleaner if instead of
doing:

token = cvar.set(...)
cvar.reset(token)

we made the API be:

token = cvar.set(...)
token.reset()

In the first version, we use 'cvar' twice, and it's a mandatory
invariant that the same ContextVar object gets used in both places;
you had to add extra code to check this and raise an error if that's
violated. It's level 5 on Rusty's scale
(http://sweng.the-davies.net/Home/rustys-api-design-manifesto)

In the second version, the ContextVar is only mentioned once, so the
invariant is automatically enforced by the API -- you can't even
express the broken version. That's level 10 on Rusty's scale, and
gives a simpler implementation too.

-n

-- 
Nathaniel J. Smith -- https://vorpus.org
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 567 v3

[Nathaniel Smith]

On Tue, Jan 16, 2018 at 5:06 PM, Yury Selivanov  wrote:
> On Tue, Jan 16, 2018 at 7:45 PM, Guido van Rossum  wrote:
>> On Tue, Jan 16, 2018 at 4:37 PM, Antoine Pitrou  wrote:
>>>
>>> On Tue, 16 Jan 2018 17:44:14 -0500
>>> Yury Selivanov  wrote:
>>>
>>> > Offloading execution to other threads
>>> > -
>>> >
>>> > It is possible to run code in a separate OS thread using a copy
>>> > of the current thread context::
>>> >
>>> > executor = ThreadPoolExecutor()
>>> > current_context = contextvars.copy_context()
>>> >
>>> > executor.submit(
>>> > lambda: current_context.run(some_function))
>>>
>>> Does it also support offloading to a separate process (using
>>> ProcessPoolExecutor in the example above)?  This would require the
>>> Context to support pickling.
>>
>>
>> I don't think that's a requirement. The transparency between the two
>> different types of executor is mostly misleading anyway -- it's like the old
>> RPC transparency problem, which was never solved IIRC. There are just too
>> many things you need to be aware of before you can successfully offload
>> something to a different process.
>
> I agree.
>
> I think it would be a very fragile thing In practice: if you have even
> one variable in the context that isn't pickleable, your code that uses
> a ProcessPool would stop working.  I would defer Context pickleability
> to 3.8+.

There's also a more fundamental problem: you need some way to match up
the ContextVar objects across the two processes, and right now they
don't have an attached __module__ or __qualname__.

I guess we could do like namedtuple and (a) capture the module where
the ContextVar was instantiated, on the assumption that that's where
it will be stored, (b) require that users pass in the name of variable
where it will be stored as the 'name' argument to ContextVar.__init__.
I tend to agree that this is something to worry about for 3.8 though.
(If we need to retrofit pickle support, we could add a
pickleable=False argument to ContextVar, and require people to pass
pickleable=True to signal that they've done the appropriate setup to
make the ContextVar identifiable across processes, and that its
contents are safe to pickle.)

-n

-- 
Nathaniel J. Smith -- https://vorpus.org
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 567 v3

[Yury Selivanov]

On Tue, Jan 16, 2018 at 7:45 PM, Guido van Rossum  wrote:
> On Tue, Jan 16, 2018 at 4:37 PM, Antoine Pitrou  wrote:
>>
>> On Tue, 16 Jan 2018 17:44:14 -0500
>> Yury Selivanov  wrote:
>>
>> > Offloading execution to other threads
>> > -
>> >
>> > It is possible to run code in a separate OS thread using a copy
>> > of the current thread context::
>> >
>> > executor = ThreadPoolExecutor()
>> > current_context = contextvars.copy_context()
>> >
>> > executor.submit(
>> > lambda: current_context.run(some_function))
>>
>> Does it also support offloading to a separate process (using
>> ProcessPoolExecutor in the example above)?  This would require the
>> Context to support pickling.
>
>
> I don't think that's a requirement. The transparency between the two
> different types of executor is mostly misleading anyway -- it's like the old
> RPC transparency problem, which was never solved IIRC. There are just too
> many things you need to be aware of before you can successfully offload
> something to a different process.

I agree.

I think it would be a very fragile thing In practice: if you have even
one variable in the context that isn't pickleable, your code that uses
a ProcessPool would stop working.  I would defer Context pickleability
to 3.8+.

Yury
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 567 v3

[Yury Selivanov]

Thanks, Victor!
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 567 v3

[Yury Selivanov]

On Tue, Jan 16, 2018 at 6:53 PM, Guido van Rossum  wrote:
> On Tue, Jan 16, 2018 at 3:26 PM, Victor Stinner 
[..]
>> I don't think that it's worth it to prevent misuage of reset(). IMHO
>> it's fine if calling reset() twice reverts the variable state twice.
>
>
> Maybe the effect of calling it twice should be specified as undefined -- the
> implementation can try to raise in simple cases.
>
> Unless Yury has a use case for the idempotency? (But with __enter__/__exit__
> as the main use case for reset() I wouldn't know what the use case for
> idempotency would be.)

I don't have any use case for idempotent reset, so I'd change it to
raise an error on second call. We can always relax this in 3.8 if
people request it to be idempotent.

Yury
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 567 v3

[Guido van Rossum]

On Tue, Jan 16, 2018 at 4:37 PM, Antoine Pitrou  wrote:

> On Tue, 16 Jan 2018 17:44:14 -0500
> Yury Selivanov  wrote:
>
> > Offloading execution to other threads
> > -
> >
> > It is possible to run code in a separate OS thread using a copy
> > of the current thread context::
> >
> > executor = ThreadPoolExecutor()
> > current_context = contextvars.copy_context()
> >
> > executor.submit(
> > lambda: current_context.run(some_function))
>
> Does it also support offloading to a separate process (using
> ProcessPoolExecutor in the example above)?  This would require the
> Context to support pickling.
>

I don't think that's a requirement. The transparency between the two
different types of executor is mostly misleading anyway -- it's like the
old RPC transparency problem, which was never solved IIRC. There are just
too many things you need to be aware of before you can successfully offload
something to a different process.

-- 
--Guido van Rossum (python.org/~guido)
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 567 v3

[Antoine Pitrou]

On Tue, 16 Jan 2018 17:44:14 -0500
Yury Selivanov  wrote:

> Offloading execution to other threads
> -
> 
> It is possible to run code in a separate OS thread using a copy
> of the current thread context::
> 
> executor = ThreadPoolExecutor()
> current_context = contextvars.copy_context()
> 
> executor.submit(
> lambda: current_context.run(some_function))

Does it also support offloading to a separate process (using
ProcessPoolExecutor in the example above)?  This would require the
Context to support pickling.

Regards

Antoine.


___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 567 v3

[Guido van Rossum]

On Tue, Jan 16, 2018 at 3:26 PM, Victor Stinner 
wrote:

> Thanks for the updated PEP v3, it's now much better than PEP v2!
>
> I have no more complain against your PEP. I vote +1 for PEP 567
> contextvars!
>

Yeah!

> The only open question I personally have is whether ContextVar.reset()
> > should be idempotent or not.  Maybe we should be strict and raise an
> > error if a user tries to reset a variable more than once with the same
> > token object?
>
> I don't think that it's worth it to prevent misuage of reset(). IMHO
> it's fine if calling reset() twice reverts the variable state twice.
>

Maybe the effect of calling it twice should be specified as undefined --
the implementation can try to raise in simple cases.

Unless Yury has a use case for the idempotency? (But with
__enter__/__exit__ as the main use case for reset() I wouldn't know what
the use case for idempotency would be.)

-- 
--Guido van Rossum (python.org/~guido)
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 567 v3

[Victor Stinner]

Hi Yury,

Thanks for the updated PEP v3, it's now much better than PEP v2!

I have no more complain against your PEP. I vote +1 for PEP 567 contextvars!

> The only open question I personally have is whether ContextVar.reset()
> should be idempotent or not.  Maybe we should be strict and raise an
> error if a user tries to reset a variable more than once with the same
> token object?

I don't think that it's worth it to prevent misuage of reset(). IMHO
it's fine if calling reset() twice reverts the variable state twice.

Victor
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

[Python-Dev] PEP 567 v3

[Yury Selivanov]

Hi,

This is a third version of PEP 567.

Changes from v2:

1. PyThreadState now references Context objects directly (instead of
referencing _ContextData).  This fixes out of sync Context.get() and
ContextVar.get().

2. Added a new Context.copy() method.

3. Renamed Token.old_val property to Token.old_value

4. ContextVar.reset(token) now raises a ValueError if the token was
created in a different Context.

5. All areas of the PEP were updated to be more precise. Context is
*no longer* defined as a read-only or an immutable mapping;
ContextVar.get() behaviour is fully defined; the immutability is only
mentioned in the Implementation section to avoid confusion; etc.

6. Added a new Examples section.

The reference implementation has been updated to include all these changes.

The only open question I personally have is whether ContextVar.reset()
should be idempotent or not.  Maybe we should be strict and raise an
error if a user tries to reset a variable more than once with the same
token object?

Other than that, I'm pretty happy with this version.  Big thanks to
everybody helping with the PEP!


PEP: 567
Title: Context Variables
Version: $Revision$
Last-Modified: $Date$
Author: Yury Selivanov 
Status: Draft
Type: Standards Track
Content-Type: text/x-rst
Created: 12-Dec-2017
Python-Version: 3.7
Post-History: 12-Dec-2017, 28-Dec-2017, 16-Jan-2018


Abstract


This PEP proposes a new ``contextvars`` module and a set of new
CPython C APIs to support context variables.  This concept is
similar to thread-local storage (TLS), but, unlike TLS, it also allows
correctly keeping track of values per asynchronous task, e.g.
``asyncio.Task``.

This proposal is a simplified version of :pep:`550`.  The key
difference is that this PEP is concerned only with solving the case
for asynchronous tasks, not for generators.  There are no proposed
modifications to any built-in types or to the interpreter.

This proposal is not strictly related to Python Context Managers.
Although it does provide a mechanism that can be used by Context
Managers to store their state.


Rationale
=

Thread-local variables are insufficient for asynchronous tasks that
execute concurrently in the same OS thread.  Any context manager that
saves and restores a context value using ``threading.local()`` will
have its context values bleed to other code unexpectedly when used
in async/await code.

A few examples where having a working context local storage for
asynchronous code is desirable:

* Context managers like ``decimal`` contexts and ``numpy.errstate``.

* Request-related data, such as security tokens and request
  data in web applications, language context for ``gettext``, etc.

* Profiling, tracing, and logging in large code bases.


Introduction


The PEP proposes a new mechanism for managing context variables.
The key classes involved in this mechanism are ``contextvars.Context``
and ``contextvars.ContextVar``.  The PEP also proposes some policies
for using the mechanism around asynchronous tasks.

The proposed mechanism for accessing context variables uses the
``ContextVar`` class.  A module (such as ``decimal``) that wishes to
use the new mechanism should:

* declare a module-global variable holding a ``ContextVar`` to
  serve as a key;

* access the current value via the ``get()`` method on the
  key variable;

* modify the current value via the ``set()`` method on the
  key variable.

The notion of "current value" deserves special consideration:
different asynchronous tasks that exist and execute concurrently
may have different values for the same key.  This idea is well-known
from thread-local storage but in this case the locality of the value is
not necessarily bound to a thread.  Instead, there is the notion of the
"current ``Context``" which is stored in thread-local storage.
Manipulation of the current context is the responsibility of the
task framework, e.g. asyncio.

A ``Context`` is a mapping of ``ContextVar`` objects to their values.
The ``Context`` itself exposes the ``abc.Mapping`` interface
(not ``abc.MutableMapping``!), so it cannot be modified directly.
To set a new value for a context variable in a ``Context`` object,
the user needs to:

* make the ``Context`` object "current" using the ``Context.run()``
  method;

* use ``ContextVar.set()`` to set a new value for the context
  variable.

The ``ContextVar.get()`` method looks for the variable in the current
``Context`` object using ``self`` as a key.

It is not possible to get a direct reference to the current ``Context``
object, but it is possible to obtain a shallow copy of it using the
``contextvars.copy_context()`` function.  This ensures that the
caller of ``Context.run()`` is the sole owner of its ``Context``
object.


Specification
=

A new standard library module ``contextvars`` is added with the
following APIs:

1. ``copy_context() -> Context`` function is used to get a copy of
   the current ``Context`` object for