On 13 August 2017 at 03:53, Yury Selivanov <yselivanov...@gmail.com> wrote: > On Sat, Aug 12, 2017 at 1:09 PM, Nick Coghlan <ncogh...@gmail.com> wrote: >> Now that you raise this point, I think it means that generators need >> to retain their current context inheritance behaviour, simply for >> backwards compatibility purposes. This means that the case we need to >> enable is the one where the generator *doesn't* dynamically adjust its >> execution context to match that of the calling function. > > Nobody *intentionally* iterates a generator manually in different > decimal contexts (or any other contexts). This is an extremely error > prone thing to do, because one refactoring of generator -- rearranging > yields -- would wreck your custom iteration/context logic. I don't > think that any real code relies on this, and I don't think that we are > breaking backwards compatibility here in any way. How many users need > about this?
I think this is a reasonable stance for the PEP to take, but the hidden execution state around the "isolated or not" behaviour still bothers me. In some ways it reminds me of the way function parameters work: the bound parameters are effectively a *shallow* copy of the passed arguments, so callers can decide whether or not they want the callee to be able to modify them based on the arguments' mutability (or lack thereof). The execution context proposal uses copy-on-write semantics for runtime efficiency, but it's essentially the same shallow copy concept applied to __next__(), send() and throw() operations (and perhaps __anext__(), asend(), and athrow() - I haven't wrapped my head around the implications for async generators and context managers yet). That similarity makes me wonder whether the "isolated or not" behaviour could be moved from the object being executed and directly into the key/value pairs themselves based on whether or not the values were mutable, as that's the way function calls work: if the argument is immutable, the callee *can't* change it, while if it's mutable, the callee can mutate it, but it still can't rebind it to refer to a different object. The way I'd see that working with an always-reverted copy-on-write execution context: 1. If a parent context wants child contexts to be able to make changes, then it should put a *mutable* object in the context (e.g. a list or class instance) 2. If a parent context *does not* want child contexts to be able to make changes, then it should put an *immutable* object in the context (e.g. a tuple or number) 3. If a child context *wants to share a context key with its parent, then it should *mutate* it in place 4. If a child context *does not* want to share a context key with its parent, then it should *rebind* it to a different object That way, instead of reverted-or-not-reverted being an all-or-nothing interpreter level decision, it can be made on a key-by-key basis by choosing whether or not to use a mutable value. To make that a little less abstract, consider a concrete example like setting a "my_web_framework.request" key: 1. The step of *setting* the key will *not* be shared with the parent context, as that modifies the underlying copy-on-write namespace, and will hence be reverted when control is passed back to the parent 2. Any *mutation* of the request object *will* be shared, since mutating the value doesn't have any effect on the copy-on-write namespace Nathaniel's example of wanting stack-like behaviour could be modeled using tuples as values: when the child context appends to the tuple, it will necessarily have to create a new tuple and rebind the corresponding key, causing the changes to be invisible to the parent context. The contextlib.contextmanager use case could then be modeled as a *separate* method that skipped the save/revert context management step (e.g. "send_with_shared_context", "throw_with_shared_context") > If someone does need this, it's possible to flip > `gi_isolated_execution_context` to `False` (as contextmanager does > now) and get this behaviour. This might be needed for frameworks like > Tornado which support coroutines via generators without 'yield from', > but I'll have to verify this. Working through this above, I think the key points that bother me about the stateful revert-or-not setting is that whether or not context reversion is desirable depends mainly on two things: - the specific key in question (indicated by mutable vs immutable values) - the intent of the code in the parent context (which could be indicated by calling different methods) It *doesn't* seem to be an inherent property of a given generator or coroutine, except insofar as there's a correlation between the code that creates generators & coroutines and the code that subsequently invokes them. > Another idea: in one of my initial PEP implementations, I exposed > gen.gi_execution_context (same for coroutines) to python as read/write > attribute. That allowed to > > (a) get the execution context out of generator (for introspection or > other purposes); > > (b) inject execution context for event loops; for instance > asyncio.Task could do that for some purpose. > > Maybe this would be useful for someone who wants to mess with > generators and contexts. Yeah, this would be useful, and could potentially avoid the need to expose a parallel set of "*_with_shared_context" methods - instead, contextlib.contextmanager could just invoke the underlying generator with an isolated context, and then set the parent context to the generator's one if it changed. > [..] >> >> def autonomous_generator(gf): >> @functools.wraps(gf) >> def wrapper(*args, **kwds): >> gi = genfunc(*args, **kwds) >> gi.gi_back = gi.gi_frame >> return gi >> return wrapper > > Nick, I still have to fully grasp the idea of `gi_back`, but one quick > thing: I specifically designed the PEP to avoid touching frames. The > current design only needs TLS and a little help from the > interpreter/core objects adjusting that TLS. It should be very > straightforward to implement the PEP in any interpreter (with JIT or > without) or compilers like Cython. I think you can just ignore that idea for now, as I've convinced myself it's orthogonal to the question of how we handle execution contexts. > [..] >> Given that, you'd have the following initial states for "revert >> context" (currently called "isolated context" in the PEP): >> >> * unawaited coroutines: true (same as PEP) >> * awaited coroutines: false (same as PEP) >> * generators (both sync & async): false (opposite of current PEP) >> * autonomous generators: true (set "gi_revert_context" or >> "ag_revert_context" explicitly) > > If generators do not isolate their context, then the example in the > Rationale section will not work as expected (or am I missing > something?). Fixing generators state leak was one of the main goals of > the PEP. Agreed - see above :) Cheers, Nick. -- Nick Coghlan | ncogh...@gmail.com | Brisbane, Australia _______________________________________________ Python-ideas mailing list Python-ideas@python.org https://mail.python.org/mailman/listinfo/python-ideas Code of Conduct: http://python.org/psf/codeofconduct/
