On 13 August 2017 at 03:53, Yury Selivanov <yselivanov...@gmail.com> wrote:
> On Sat, Aug 12, 2017 at 1:09 PM, Nick Coghlan <ncogh...@gmail.com> wrote:
>> Now that you raise this point, I think it means that generators need
>> to retain their current context inheritance behaviour, simply for
>> backwards compatibility purposes. This means that the case we need to
>> enable is the one where the generator *doesn't* dynamically adjust its
>> execution context to match that of the calling function.
> Nobody *intentionally* iterates a generator manually in different
> decimal contexts (or any other contexts). This is an extremely error
> prone thing to do, because one refactoring of generator -- rearranging
> yields -- would wreck your custom iteration/context logic. I don't
> think that any real code relies on this, and I don't think that we are
> breaking backwards compatibility here in any way. How many users need
> about this?

I think this is a reasonable stance for the PEP to take, but the
hidden execution state around the "isolated or not" behaviour still
bothers me.

In some ways it reminds me of the way function parameters work: the
bound parameters are effectively a *shallow* copy of the passed
arguments, so callers can decide whether or not they want the callee
to be able to modify them based on the arguments' mutability (or lack

The execution context proposal uses copy-on-write semantics for
runtime efficiency, but it's essentially the same shallow copy concept
applied to __next__(), send() and throw() operations (and perhaps
__anext__(), asend(), and athrow() - I haven't wrapped my head around
the implications for async generators and context managers yet).

That similarity makes me wonder whether the "isolated or not"
behaviour could be moved from the object being executed and directly
into the key/value pairs themselves based on whether or not the values
were mutable, as that's the way function calls work: if the argument
is immutable, the callee *can't* change it, while if it's mutable, the
callee can mutate it, but it still can't rebind it to refer to a
different object.

The way I'd see that working with an always-reverted copy-on-write
execution context:

1. If a parent context wants child contexts to be able to make
changes, then it should put a *mutable* object in the context (e.g. a
list or class instance)
2. If a parent context *does not* want child contexts to be able to
make changes, then it should put an *immutable* object in the context
(e.g. a tuple or number)
3. If a child context *wants to share a context key with its parent,
then it should *mutate* it in place
4. If a child context *does not* want to share a context key with its
parent, then it should *rebind* it to a different object

That way, instead of reverted-or-not-reverted being an all-or-nothing
interpreter level decision, it can be made on a key-by-key basis by
choosing whether or not to use a mutable value.

To make that a little less abstract, consider a concrete example like
setting a "my_web_framework.request" key:

1. The step of *setting* the key will *not* be shared with the parent
context, as that modifies the underlying copy-on-write namespace, and
will hence be reverted when control is passed back to the parent
2. Any *mutation* of the request object *will* be shared, since
mutating the value doesn't have any effect on the copy-on-write

Nathaniel's example of wanting stack-like behaviour could be modeled
using tuples as values: when the child context appends to the tuple,
it will necessarily have to create a new tuple and rebind the
corresponding key, causing the changes to be invisible to the parent

The contextlib.contextmanager use case could then be modeled as a
*separate* method that skipped the save/revert context management step
(e.g. "send_with_shared_context", "throw_with_shared_context")

> If someone does need this, it's possible to flip
> `gi_isolated_execution_context` to `False` (as contextmanager does
> now) and get this behaviour. This might be needed for frameworks like
> Tornado which support coroutines via generators without 'yield from',
> but I'll have to verify this.

Working through this above, I think the key points that bother me
about the stateful revert-or-not setting is that whether or not
context reversion is desirable depends mainly on two things:

- the specific key in question (indicated by mutable vs immutable values)
- the intent of the code in the parent context (which could be
indicated by calling different methods)

It *doesn't* seem to be an inherent property of a given generator or
coroutine, except insofar as there's a correlation between the code
that creates generators & coroutines and the code that subsequently
invokes them.

> Another idea: in one of my initial PEP implementations, I exposed
> gen.gi_execution_context (same for coroutines) to python as read/write
> attribute. That allowed to
> (a) get the execution context out of generator (for introspection or
> other purposes);
> (b) inject execution context for event loops; for instance
> asyncio.Task could do that for some purpose.
> Maybe this would be useful for someone who wants to mess with
> generators and contexts.

Yeah, this would be useful, and could potentially avoid the need to
expose a parallel set of "*_with_shared_context" methods - instead,
contextlib.contextmanager could just invoke the underlying generator
with an isolated context, and then set the parent context to the
generator's one if it changed.

> [..]
>>     def autonomous_generator(gf):
>>         @functools.wraps(gf)
>>         def wrapper(*args, **kwds):
>>             gi = genfunc(*args, **kwds)
>>             gi.gi_back = gi.gi_frame
>>             return gi
>>         return wrapper
> Nick, I still have to fully grasp the idea of `gi_back`, but one quick
> thing: I specifically designed the PEP to avoid touching frames. The
> current design only needs TLS and a little help from the
> interpreter/core objects adjusting that TLS. It should be very
> straightforward to implement the PEP in any interpreter (with JIT or
> without) or compilers like Cython.

I think you can just ignore that idea for now, as I've convinced
myself it's orthogonal to the question of how we handle execution

> [..]
>> Given that, you'd have the following initial states for "revert
>> context" (currently called "isolated context" in the PEP):
>> * unawaited coroutines: true (same as PEP)
>> * awaited coroutines: false (same as PEP)
>> * generators (both sync & async): false (opposite of current PEP)
>> * autonomous generators: true (set "gi_revert_context" or
>> "ag_revert_context" explicitly)
> If generators do not isolate their context, then the example in the
> Rationale section will not work as expected (or am I missing
> something?). Fixing generators state leak was one of the main goals of
> the PEP.

Agreed - see above :)


Nick Coghlan   |   ncogh...@gmail.com   |   Brisbane, Australia
Python-ideas mailing list
Code of Conduct: http://python.org/psf/codeofconduct/

Reply via email to