Re: [Python-ideas] PEP 550 v2

[Nick Coghlan]

On 17 August 2017 at 01:22, Yury Selivanov  wrote:
> On Wed, Aug 16, 2017 at 4:07 AM, Nick Coghlan  wrote:
>>> Coroutine Object Modifications
>>> ^^
>>>
>>> To achieve this, a small set of modifications to the coroutine object
>>> is needed:
>>>
>>> * New ``cr_local_context`` attribute.  This attribute is readable
>>>   and writable for Python code.
>>
>> For ease of introspection, it's probably worth using a common
>> `__local_context__` attribute name across all the different types that
>> support one, and encouraging other object implementations to do the
>> same.
>>
>> This isn't like cr_await and gi_yieldfrom, where we wanted to use
>> different names because they refer to different kinds of objects.
>
> We also have cr_code and gi_code, which are used for introspection
> purposes but refer to CodeObject.

Right, hence https://bugs.python.org/issue31230 :)

(That suggestion is prompted by the fact that if we'd migrated gi_code
to __code__ in 3.0, the same way we migrated func_code, then cr_code
and ag_code would almost certainly have followed the same
dunder-naming convention, and
https://github.com/python/cpython/pull/3077 would never have been
necessary)

> I myself don't like the mess the C-style convention created for our
> Python code (think of what the "dis" and "inspect" modules have to go
> through), so I'm +0 for having "__local_context__".

I'm starting to think this should be __private_context__ (to convey
the *intent* of the attribute), rather than naming it after the type
that it's expected to store.

Thinking about this particular attribute name did prompt the question
of how we want PEP 550 to interact with the exec builtin, though, as
well as raising some questions around a number of other code execution
cases:

1. What is the execution context for top level code in a module?
2. What is the execution context for the import machinery in an import
statement?
3. What is the execution context for the import machinery when invoked
via importlib?
4. What is the execution context for the import machinery when invoked
via the C API?
5. What is the execution context for the import machinery when invoked
via the runpy module?
6. What is the execution context for things like the timeit module,
templating engines, etc?
7. What is the execution context for codecs and codec error handlers?
8. What is the execution context for __del__ methods and weakref callbacks?
9. What is the execution context for trace hooks and other really low
level machinery?
10. What is the execution context for displayhook and excepthook?

I think a number of those (top level module code executed via the
import system, the timeit module, templating engines) can be addressed
by saying that the exec builtin always creates a completely fresh
execution context by default (with no access to the parent's execution
context), and will gain a new keyword-only parameter that allows you
to specify an execution context to use. That way, exec'ed code will be
independent by default, but users of exec() will be able to opt in to
handing it like a normal function call by passing in the current
context. The default REPL, the code module and the IDLE shell window
would need to be updated so that they use a shared context for
evaluating the user supplied code snippets, while keeping their own
context separate.

While top-level code would always run in a completely fresh context
for imports, the runpy module would expose the same setting as the
exec builtin, so the executed code would be isolated by default, but
you could opt in to using a particular execution context if you wanted
to.

Codecs and codec error handlers I think will be best handled in a way
similar to generators, where they have their own private context (so
they can't alter the caller's context), but can *read* the caller's
context (so the context can be used as a way of providing
context-dependent codec settings).

That "read-only" access model also feels like the right option for the
import machinery - regardless of whether it's accessed via the import
statement, importlib, the C API, or the runpy module, the import
machinery should be able to *read* the dynamic context, but not make
persistent changes to it.

Since they can be executed at arbitrary points in the code, it feels
to me that __del__ methods and weakref callbacks should *always* be
executed in a completely pristine execution context, with no access
whatsoever to any thread's dynamic context.

I think we should leave the execution context alone for the really low
level hooks, and simply point out that yes, these have the ability to
do weird things to the execution context, just as they have the power
to do weird things to local variables, so they need to be handles with
care.

For displayhook and excepthook, I don't have a particularly strong
intuition, so my default recommendation would be the read-only access
proposed for generators, 

Re: [Python-ideas] PEP 550 v2

[Nick Coghlan]

On 17 August 2017 at 02:55, Yury Selivanov  wrote:
> And immediately after I hit "send" I realized that this is a bit more
> complicated.
>
> In order for Tasks to remember the full execution context of where
> they were created, we need a new method that would allow to run with
> *both* exec and local contexts:
>
> class Task:
>
>def __init__(self, coro):
>...
>self.local_context = sys.new_local_context()
>self.exec_context = sys.get_execution_context()
>
>def step():
>
>  sys.run_with_contexts(self.exec_context, self.local_context,
> self.coro.send)

I don't think that's entirely true, since you can nest the calls even
without a combined API:

sys.run_with_execution_context(self.exec_context,
sys.run_with_local_context, self.local_context, self.coro.send)

Offering a combined API may still make sense for usability and
efficiency reasons, but it isn't strictly necessary.

Cheers,
Nick.

-- 
Nick Coghlan   |   ncogh...@gmail.com   |   Brisbane, Australia
___
Python-ideas mailing list
Python-ideas@python.org
https://mail.python.org/mailman/listinfo/python-ideas
Code of Conduct: http://python.org/psf/codeofconduct/

Re: [Python-ideas] PEP 550 v2

[Nick Coghlan]

On 17 August 2017 at 04:38, Yury Selivanov  wrote:
> On Wed, Aug 16, 2017 at 1:13 PM, Stefan Krah  wrote:
> While I'm trying to avoid using scoping terminology for PEP 550, there's
> one parallel -- as with regular Python scoping you have global variables
> and you have local variables.
>
> You can use the locals() to access to your local scope, and you can use
> globals() to access to your global scope.

To be honest, the difference between LocalContext and ExecutionContext
feels more like the difference between locals() and lexical closure
variables than it does the difference between between locals() and
globals().

It's just that where the scoping rules are a compile time thing
related to lexical closures, PEP 550 is about defining a dynamic
context.

> Similarly in PEP 550, you have your LocalContext and ExecutionContext.
> We don't want to call ExecutionContext a "Global Context" because
> it is fundamentally OS-thread-specific (contrary to Python globals).

In addition to it being different from the way the decimal module
already uses the phrase, one of the reasons I don't want to call it a
LocalContext is because doing so brings in the suggestion that it is
somehow connected to the locals() scope, and it isn't - there are
plenty of things (most notably, function calls) that will change the
active local namespace, but *won't* change the active execution
context.

> LocalContexts are created for threads, generators, coroutines and are
> really similar to local scoping.  Adding more names for local contexts
> like CoroutineLocalContext, GeneratorLocalContext won't solve anything
> either.  All in all, Local Context is what its name stands for -- it's a
> local context for your current logical scope, be it a coroutine or a
> generator.

But unlike locals() itself, it *isn't* linked to a specific frame of
execution - it's deliberately designed to be shared *between* frames.

If you don't like either of the ExecutionContext/ExecutionEnvironment
or ExecutionContext/ExecutionContextChain combinations, how would you
feel about ExecutionContext + DynamicContext?

Saying that "ck.set_value(value) sets the value corresponding to the
given context key in the currently active execution context" is still
my preferred terminology for setting values, and I think the following
would work well for reading values:

ck.get_value() attempts to look up the value for that key in the
currently active execution context.
If it doesn't find one, it then tries each of the execution
contexts in the currently active dynamic context.
If it *still* doesn't find one, then it will set the default value
in the outermost execution context and then return that value.

One thing I like about that phrasing is that we'd be using the word
dynamic in exactly the same sense that dynamic scoping uses it, and
the dynamic context mechanism would become PEP 550's counterpart to
the lexical closure support in Python's normal scoping rules.

Cheers,
Nick.

-- 
Nick Coghlan   |   ncogh...@gmail.com   |   Brisbane, Australia
___
Python-ideas mailing list
Python-ideas@python.org
https://mail.python.org/mailman/listinfo/python-ideas
Code of Conduct: http://python.org/psf/codeofconduct/

Re: [Python-ideas] PEP 550 v2

[Nick Coghlan]

On 17 August 2017 at 02:36, Yury Selivanov  wrote:
> Yeah, this is tricky. The main issue is indeed the confusion of what
> methods you need to call -- "get/set" or
> "get_local_state/set_local_state".
>
> On some level the problem is very similar to regular Python scoping rules:
>
> 1. we have local hames
> 2. we have global names
> 3. we nave 'nonlocal' modifier
>
> IOW scoping isn't easy, and you need to be conscious of what you do.
> It's just that we are so used to these scoping rules that they have a
> low cognitive effort for us.
>
> One of the ideas that I have in mind is to add another level of
> indirection to separate "global get" from "local set/get":
>
> 1. Rename ContextItem to ContextKey (reasoning for that in parallel thread)
>
> 2. Remove ContextKey.set() method
>
> 3. Add a new ContextKey.value() -> ContextValue
>
> ck = ContextKey()
>
> with ck.value() as val:
> val.set(spam)
> yield
>
> or
>
>  val = ck.value()
>  val.set(spam)
>  try:
>   yield
>  finally:
>   val.clear()
>
> Essentially ContextValue will be the only API to set values in
> execution context. ContextKey.get() will be used to get them.
>
> Nathaniel, Nick, what do you guys think?

I think I don't want to have try to explain to anyone what happens if
I get a context value in my current execution environment and then
send that value reference into a different execution context :)

So I'd prefer my earlier proposal of:

# Resolve key in current execution environment
ck.get_value()
# Assign to key in current execution context
ck.set_value(value)
# Assign to key in specific execution context
sys.run_with_active_context(ec, ck.set_value, value)

One suggestion I do like is Stefan's one of using "ExecutionContext"
to refer to the namespace that ck.set_value() writes to, and then
"ExecutionEnvironment" for the whole chain that ck.get_value() reads.

Similar to "generator" and "package", we'd still end up with "context"
being inherently ambiguous when used without qualification:

- PEP 550 execution context
- exception handling context (for chained exceptions)
- with statement context
- various context objects, like the decimal context

But we wouldn't have two different kinds of context within PEP 550
itself. Instead, we'd have to start disambiguating the word
environment:

- PEP 550 execution environment
- process environment (i.e. os.environ)

The analogy between process environments and execution environments
wouldn't be exact (since the key-value pairs in process environments
are copied eagerly rather than via lazily chained lookups), but once
you account for that, the parallels between an operating system level
process environment tree and a Python level execution environment tree
as proposed in PEP 550 seem like they would be helpful rather than
confusing.

> [..]
>>> * ``sys.get_execution_context()`` function.  The function returns a
>>>   copy of the current EC: an ``ExecutionContext`` instance.
>>
>> If there are enough of these functions then it might make sense to
>> stick them in their own module instead of adding more stuff to sys. I
>> guess worrying about that can wait until the API details are more firm
>> though.
>
> I'm OK with this idea -- pystate.c becomes way too crowded.
>
> Maybe we should just put this stuff in _contextlib.c and expose in the
> contextlib module.

Yeah, I'd be OK with that - if we're going to reuse the word, it makes
sense to reuse the module to expose the related machinery.

That said, if we do go that way *and* we decide to offer a
coroutine-only backport, I see an offer of contextlib2
co-maintainership in your future ;)

>>>   * If ``coro.cr_local_context`` is an empty ``LocalContext`` object
>>> that ``coro`` was created with, the interpreter will set
>>> ``coro.cr_local_context`` to ``None``.
>>
>> I like all the ideas in this section, but this specific point feels a
>> bit weird. Coroutine objects need a second hidden field somewhere to
>> keep track of whether the object they end up with is the same one they
>> were created with?
>
> Yes, I planned to have a second hidden field, as Coroutines will have
> their cr_local_context set to NULL, and that will be their empty LC.
> So a second internal field is needed to disambiguate NULL -- meaning
> an "empty context" and NULL meaning "use outside local context".
>
> I omitted this from the PEP to make it a bit easier to digest, as this
> seemed to be a low-level implementation detail.

Given that the field is writable, I think it makes more sense to just
choose a suitable default, and then rely on other code changing that
default when its not right.

For generators: set it to an empty context by default, have
contextlib.contextmanager (and similar wrapper) clear it
For coroutines: set it to None by default, have async task managers
give top level coroutines their own private context

No hidden flags, no magic value