Re: [Python-Dev] Scope, not context? (was Re: PEP 550 v3 naming)
On 24 August 2017 at 23:52, Barry Warsaw wrote: > Guido van Rossum wrote: >> On Tue, Aug 22, 2017 at 7:12 PM, Nathaniel Smith wrote: >> >> I worry that that's going to lead more people astray thinking this has >> something to do with contextlib, which it really doesn't (it's much more >> closely related to threading.local()). > > This is my problem with using "Context" for this PEP. Although I can't > keep up with all names being thrown around, it seems to me that in > Python we already have a well-established meaning for "contexts" -- > context managers, and the protocols they implement as they participate > in `with` statements. We have contextlib which reinforces this. What's > being proposed in PEP 550 is so far removed from this concept that I > think it's just going to cause confusion (well, it does in me anyway!). While I understand the concern, I think context locals and contextlib are more closely related than folks realise, as one of the main problems that the PEP is aiming to solve is that with statements (and hence context managers) *do not work as expected* when their body includes "yield", "yield from" or "await" . The reason they don't work reliably is because in the absence of frame hacks, context managers are currently limited to manipulating process global and thread local state - there's no current notion of context local storage that interacts nicely with the frame suspension keywords. And hence in the absence of any support for context local state, any state changes made by context managers tend to remain in effect *even if the frame that entered the context manager gets suspended*. Context local state makes it possible to solve that problem, as it means that encountering one of the frame suspension keywords in the body of a with statement will implicitly revert all changes made to context local variables until such time as that particular frame is resumed, and you have to explicitly opt-in on a per-instance basis to instead allow those state changes to affect the calling context that's suspending & resuming the frame. That said, I can also see Guido's point that the proposed new APIs have a very different flavour to the existing APIs in contextlib, so it doesn't necessarily make sense to use that module directly. Building on the "context locals" naming scheme I've been discussing elsewhere, one possible name for a dedicated module would be "contextlocals" giving: # Read/write access to individual context locals * context_var = contextlocals.new_context_local(name: str='...') # Context local storage manipulation * context = contextlocals.new_context_local_state() * contextlocals.run_with_context_locals(context: ContextLocalState, func, *args, **kwargs). * __context_locals__ attribute on generators and coroutines # Overall execution context manipulation * current_ec = contextlocals.get_execution_context() * new_ec = contextlocals.new_execution_context() * contextlocals.run_with_execution_context(ec: ExecutionContext, func, *args, **kwargs). The remaining connection with "contextlib" would be that @contextlib.contextmanager (and its async counterpart) would implicitly clear the __context_locals__ attribute on the underlying generator instance so that context local state changes made there will affect the context where the context manager is actually being used. Cheers, Nick. -- Nick Coghlan | ncogh...@gmail.com | Brisbane, Australia ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 550 v3 naming
I think we should let the naming issue go for now, while Yury et al. are rewriting the PEP. We'll revisit it after we're more comfortable with the semantics. On Thu, Aug 24, 2017 at 9:55 PM, Nick Coghlan wrote: > On 25 August 2017 at 01:00, Guido van Rossum wrote: > > It shouldn't be called a namespace unless the dominant access is via > > attributes. > > That makes sense. > > Since the main purpose of that part of the Python API is to offer an > opaque handle to where the context locals store their values, > something semantically neutral like "State" may work: > > - ContextLocal: read/write API > - ContextLocalState: where ContextLocal instances actually store things > - ExecutionContext: nested stack of context local states > > The attribute on generators and coroutines could then be called > "__context_locals__", and that would either be None (indicating that > any context local references will just use the already active context > local storage), or else it would be set to a ContextLocalState > instance (indicate that starting & stopping the operation will push & > pop the given context local state). > > Cheers, > Nick. > > -- > Nick Coghlan | ncogh...@gmail.com | Brisbane, Australia > -- --Guido van Rossum (python.org/~guido) ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 550 v3 naming
On 25 August 2017 at 01:00, Guido van Rossum wrote: > It shouldn't be called a namespace unless the dominant access is via > attributes. That makes sense. Since the main purpose of that part of the Python API is to offer an opaque handle to where the context locals store their values, something semantically neutral like "State" may work: - ContextLocal: read/write API - ContextLocalState: where ContextLocal instances actually store things - ExecutionContext: nested stack of context local states The attribute on generators and coroutines could then be called "__context_locals__", and that would either be None (indicating that any context local references will just use the already active context local storage), or else it would be set to a ContextLocalState instance (indicate that starting & stopping the operation will push & pop the given context local state). Cheers, Nick. -- Nick Coghlan | ncogh...@gmail.com | Brisbane, Australia ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Scope, not context? (was Re: PEP 550 v3 naming)
Barry Warsaw wrote: I actually think Python’s scoping rules are fairly easy to grasp, The problem is that the word "scope", as generally used in relation to programming languages, has to do with visibility of names. A variable is "in scope" at a particular point in the code if you can acccess it just by writing its name there. The things we're talking about are never "in scope" in that sense. What we have is something similar to a dynamic scope, but a special action is required to access bindings in it. I can't think of any established term for things like that. The closest I'ves seen is one dialect of Scheme that called it a "fluid environment". In that dialect, fluid-let didn't create bindings in the normal scope, and you had to use specific functions to access them. -- Greg ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Scope, not context? (was Re: PEP 550 v3 naming)
Barry Warsaw wrote: This is my problem with using "Context" for this PEP. Although I can't keep up with all names being thrown around, Not sure whether it helps, but a similar concept in some Scheme dialects is called a "fluid binding". e.g. Section 5.4 of http://www.scheme.com/csug8/binding.html It's not quite the same thing as we're talking about here (the names bound exist in the normal name lookup scope rather than a separate namespace) but maybe some of the terminology could be adapted. -- Greg ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 550 v3 naming
On Thu, Aug 24, 2017 at 5:37 AM, Nick Coghlan wrote: > Migrating a (variant of a) naming subthread from python-ideas over to > here, does the following sound plausible to anyone else?: > > ContextLocal - read/write access API (via get()/set() methods) > ContextLocalNamespace - active mapping that CL.get()/set() manipulates > ExecutionContext - current stack of context local namespaces Overall it makes sense. The following derivative might be more clear: ContextLocal ContextLocals ChainedContextLocals (or some variation explicitly calling out the chaining) Alternatively, similar to other suggestions: ContextVar ContextVars ChainedContextVars I also think "state" helps disambiguate "context": ContextVar ContextState ChainedContextState or: ContextVar ContextVars ContextState (perhaps a little too ambiguous without the explicit "chained") -eric ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Scope, not context? (was Re: PEP 550 v3 naming)
On Thu, 24 Aug 2017 17:06:09 -0400 Barry Warsaw wrote: > On Aug 24, 2017, at 16:01, francismb wrote: > > On 08/24/2017 03:52 PM, Barry Warsaw wrote: > >> Guido van Rossum wrote: > >>> On Tue, Aug 22, 2017 at 7:12 PM, Nathaniel Smith wrote: > >>> > >>> I worry that that's going to lead more people astray thinking this has > >>> something to do with contextlib, which it really doesn't (it's much more > >>> closely related to threading.local()). > >> > > If it's hard to find a name due collision with related meanings or > > already taken buckets then I would suggest just inventing a new word. An > > artificial one. > > > > What do you thing? > > I propose RaymondLuxuryYach_t, but we’ll have to pronounce it > ThroatwobblerMangrove. Sorry, but I think we should prononce it ThroatwobblerMangrove. Regards Antoine. ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Scope, not context? (was Re: PEP 550 v3 naming)
On Aug 24, 2017, at 16:01, francismb wrote: > On 08/24/2017 03:52 PM, Barry Warsaw wrote: >> Guido van Rossum wrote: >>> On Tue, Aug 22, 2017 at 7:12 PM, Nathaniel Smith wrote: >>> >>> I worry that that's going to lead more people astray thinking this has >>> something to do with contextlib, which it really doesn't (it's much more >>> closely related to threading.local()). >> > If it's hard to find a name due collision with related meanings or > already taken buckets then I would suggest just inventing a new word. An > artificial one. > > What do you thing? I propose RaymondLuxuryYach_t, but we’ll have to pronounce it ThroatwobblerMangrove. import-dinsdale-ly y’rs, -Barry signature.asc Description: Message signed with OpenPGP ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Scope, not context? (was Re: PEP 550 v3 naming)
Hi, On 08/24/2017 03:52 PM, Barry Warsaw wrote: > Guido van Rossum wrote: >> On Tue, Aug 22, 2017 at 7:12 PM, Nathaniel Smith wrote: >> >> I worry that that's going to lead more people astray thinking this has >> something to do with contextlib, which it really doesn't (it's much more >> closely related to threading.local()). > If it's hard to find a name due collision with related meanings or already taken buckets then I would suggest just inventing a new word. An artificial one. What do you thing? Thanks, --francis ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] python issue27584 AF_VSOCK support
On 08/17/2017 05:38 PM, Christian Heimes wrote: On 2017-08-17 14:21, Cathy Avery wrote: Hi, I have a python pull request https://github.com/python/cpython/pull/2489 that introduces AF_VSOCK to the python C socket module. Unfortunately things have gone quiet for several weeks. Since I am new to this process I am unclear on what to expect as the next steps. I believe I have addressed all issues. If I am mistaken please let me know. A real functioning test setup is a bit painful and I would be happy to help in anyway I can. The Microsoft Hyper V team is now introducing support of vsock in the upstream linux kernel so its adoption is spreading amongst the various hypervisors. Any guidance on the basic timelines would be very helpful. Hi Cathy, thanks for your contribution! Your patch looks mostly fine. I pointed out some minor issues in my review. Regards, Christian Thanks! I'm going to push it soon. I think I got everything. ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Scope, not context? (was Re: PEP 550 v3 naming)
On Thu, Aug 24, 2017 at 8:23 AM, Yury Selivanov wrote: > Contrary to scoping, the programmer is much less likely to deal with > Execution Context. How often do we use "threading.local()"? This is an important point. PEP 550 is targeting library authors, right? Most folks will not be interacting with the new functionality except perhaps indirectly via context managers. So I'm not convinced that naming collisions with more commonly used concepts is as much a concern, particularly because of conceptual similarity. However, if the functionality of the PEP would be used more commonly then the naming would be a stronger concern. -eric ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Scope, not context? (was Re: PEP 550 v3 naming)
On Thu, Aug 24, 2017 at 7:52 AM, Barry Warsaw wrote: > Guido van Rossum wrote: >> On Tue, Aug 22, 2017 at 7:12 PM, Nathaniel Smith wrote: >> >> I worry that that's going to lead more people astray thinking this has >> something to do with contextlib, which it really doesn't (it's much more >> closely related to threading.local()). > > This is my problem with using "Context" for this PEP. Although I can't > keep up with all names being thrown around, it seems to me that in > Python we already have a well-established meaning for "contexts" -- > context managers, and the protocols they implement as they participate > in `with` statements. We have contextlib which reinforces this. What's > being proposed in PEP 550 is so far removed from this concept that I > think it's just going to cause confusion (well, it does in me anyway!). The precedent (and perhaps inspiration) here lies in decimal.context [1] and ssl.SSLContext [2]. email.Policy [3] qualifies in spirit, as does the import state [4]. Each encapsulates the state of some subsystem. They can be enabled in the current thread via a context manager. (Well, only the first two, but the latter two are strong candidates.) They are each specific to a *logical* execution context. Most notably, each is implicit global state in the current thread of execution. PEP 550, if I've understood right, is all about supporting these contexts in other logical threads of execution than just Python/OS threads (e.g. async, generators). Given all that, "context" is an understandable name to use here. Personally, I'm still on the fence on if "context" fits in the name. :) None of the proposed names have felt quite right thus far, which is probably why we're still talking about it! :) > > To me, the functionality proposed in PEP 550 feels more like a "scope" > than a "context". Unlike a lexical scope, it can't be inferred from the > layout of the source code. It's more of a dynamic "execution scope" and > the stacking of "local execution scopes" reinforces that for me. You > use a key to find a value in the current execution scope, and it chains > up until the key is found or you've reached the top of the local > execution (defined as the thread start, etc.). "scope" fits because it's all about chained lookup in implicit namespaces. However, to me the focus of PEP 550 is on the context (encapsulated state) more than the chaining. > > One other suggestion: maybe we shouldn't put these new functions in sys, > but instead put them in their own module? It feels analogous to the gc > module; all those functions could have gone in sys since they query and > effect the Python runtime system, but it makes more sense (and improves > the naming) by putting them in their own module. It also segregates the > functionality so that sys doesn't become a catchall that overloads you > when you're reading through the sys module documentation. +1 -eric [1] https://docs.python.org/3/library/ssl.html#ssl.SSLContext [2] https://docs.python.org/3/library/decimal.html#context-objects [3] https://docs.python.org/3/library/email.policy.html [4] https://www.python.org/dev/peps/pep-0406/ ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Scope, not context? (was Re: PEP 550 v3 naming)
On 08/24/2017 06:52 AM, Barry Warsaw wrote: To me, the functionality proposed in PEP 550 feels more like a "scope" than a "context". Unlike a lexical scope, it can't be inferred from the layout of the source code. It's more of a dynamic "execution scope" and the stacking of "local execution scopes" reinforces that for me. You use a key to find a value in the current execution scope, and it chains up until the key is found or you've reached the top of the local execution (defined as the thread start, etc.). Scope and dynamic certainly feel like the important concepts in PEP 550, and Guido (IIRC) has already said these functions do not belong in contextlib, so context may not be a good piece of the name. Some various ideas: - ExecutionScope, LocalScope - DynamicScope, ScopeLocals - DynamicExecutionScope, DynamicLocals - DynamicExecutionScope, ExecutionLocals - DynamicExecutionScope, ScopeLocals -- ~Ethan~ ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Scope, not context? (was Re: PEP 550 v3 naming)
On Thu, Aug 24, 2017 at 10:38 AM, Barry Warsaw wrote: [..] I'll snip the naming discussion for now, I'm really curious to see what other people will say. > A different tack would more closely align with PEP 550’s heritage in > thread-local storage, calling these things “execution storage”. I think I > read Guido suggest elsewhere using a namespace here so that in common code > you’d only have to change the “threading.local()” call to migrate to PEP 550. > It might be neat if you could do something like: > > import execution > els = execution.local() > els.x = 1 A couple of relevant updates on this topic in old python-ideas threads (I'm not sure you've seen them): https://mail.python.org/pipermail/python-ideas/2017-August/046888.html https://mail.python.org/pipermail/python-ideas/2017-August/046889.html Unfortunately it's not feasible to re-use the "local()" idea for PEP 550. The "local()" semantics imposes many constraints and complexities to the design, while offering only "users already know it" argument for it. And even if we had "execution.local()" API, it would not be safe to always replace every "theading.local()" with it. Sometimes what you need is an actual TLS. Sometimes your design actually depends on it, even if you are not aware of that. Updating existing libraries to use PEP 550 should be a very conscious decision, simply because it has different semantics/guarantees. Y ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 550 leak-in vs leak-out, why not just a ChainMap
On Thu, Aug 24, 2017 at 10:05 AM, Jim J. Jewett wrote:
> On Thu, Aug 24, 2017 at 1:12 AM, Yury Selivanov > On Thu, Aug 24, 2017
> at 12:32 AM, Jim J. Jewett wrote:
>
>> The key requirement for using immutable datastructures is to make
>> "get_execution_context" operation fast.
>
> Do you really need the whole execution context, or do you just need
> the current value of a specific key? (Or, sometimes, the writable
> portion of the context.)
We really need a snapshot of all keys/values. If we have the following "chain":
EC = [{'a': 'b'}, {'spam': 'ham'}, {}, {'a': 'foo', 'num': 42}]
we need sys.get_execution_context() to return this:
{'spam': 'ham', 'a': 'foo', 'num': 42}
To do this with chain map and mutable maps, you need to traverse the
whole chain and merge all dicts -- this is very expensive. Immutable
datastructures make it possible to avoid merge and traverse (in most
cases) so the operation becomes cheap.
Don't forget, that the EC is a *dynamic* stack -- a generator starts,
it pushes its LogicalContext, etc.
Without a way to get a full "snapshot" of the EC we can't support
asynchronous tasks:
If you look at this small example:
foo = new_context_key()
async def nested():
await asyncio.sleep(1)
print(foo.get())
async def outer():
foo.set(1)
await nested()
foo.set(1000)
l = asyncio.get_event_loop()
l.create_task(outer())
l.run_forever()
If will print "1", as "nested()" coroutine will see the "foo" key when
it's awaited.
Now let's say we want to refactor this snipped and run the "nested()"
coroutine with a timeout:
foo = new_context_key()
async def nested():
await asyncio.sleep(1)
print(foo.get())
async def outer():
foo.set(1)
await asyncio.wait_for(nested(), 10) # !!!
foo.set(1000)
l = asyncio.get_event_loop()
l.create_task(outer())
l.run_forever()
So we wrap our `nested()` in a `wait_for()`, which creates a new
asynchronous tasks to run `nested()`. That task will now execute on
its own, separately from the task that runs `outer()`. So we need to
somehow capture the full EC at the moment `wait_for()` was called, and
use that EC to run `nested()` within it. If we don't do this, the
refactored code would print "1000", instead of "1".
If asynchronous tasks topic is not enough: suppose you want to write a
wrapper on top of concurrent.futures.ThreadPoolExecutor to offload
some calculation to a threadpool. If you can "snapshot" the EC, you
can do it: when you schedule a job, the EC is captured, and it is a
guarantee that it will not be mutated before the job is executed. And
it's also guaranteed that the job will not randomly mutate the EC of
the code that scheduled it.
>> Currently, the PEP doesn't do
>> a good job at explaining why we need that operation and why it will be
>> used by asyncio.Task and call_soon, so I understand the confusion.
>
> OK, the schedulers need the whole context, but if implemented as a
> ChainMap (instead of per-key),
I think you are a bit confused here (or I don't follow you) with the
"per-key" ConceptKey design.
With ConceptKeys:
key = new_concept_key('spam')
key.set('a')
# EC = [..., {key: 'a'}]
without:
ec.set('key', 'a')
# EC = [..., {'key': 'a'}]
ConceptKey is an extra indirection layer that allows for granular
caching and avoids the names clashing.
> isn't that just a single constant? As
> in, don't they always schedule to the same thread? And when they need
> another map, isn't that because the required context is already
> available from whichever code requested the scheduling?
>
>>> (A) How many values do you expect a typical generator to use? The
>>> django survey suggested mostly 0, sometimes 1, occasionally 2. So
>>> caching the values of all possible keys probably won't pay off.
>
>> Not many, but caching is still as important, because some API users
>> want the "get()" operation to be as fast as possible under all
>> conditions.
>
> Sure, but only because they view it as a hot path; if the cost of that
> speedup is slowing down another hot path, like scheduling the
> generator in the first place, it may not be worth it.
>
> According to the PEP timings, HAMT doesn't beat a copy-on-write dict
> until over 100 items, and never beats a regular dict.That suggests
> to me that it won't actually help the overall speed for a typical (as
> opposed to worst-case) process.
It's a bit more complicated, unfortunately.
The "100 items" it true when we can just memcpy all keys/values of the
dict when we want to clone it. I've proposed a patch to do that, and
it works.
The old implementation of "dict.copy()" created a new dict, iterated
over items of the old dict, and inserted them to the new one. This 5x
slower than memcpy.
The subtle difference is that the memcpy implementation does not
resize the dict. The resize is needed when we delete items from it
(which will happen with
Re: [Python-Dev] PEP 550 v3 naming
It shouldn't be called a namespace unless the dominant access is via
attributes.
On Thu, Aug 24, 2017 at 4:37 AM, Nick Coghlan wrote:
> On 24 August 2017 at 08:47, Ethan Furman wrote:
> >
> > ContextVars is actually a different name for LogicalContext. So it would
> > be:
> >
> > ExecutionContext = [ContextVars()[, ContextVars()[ ...]]]
> >
> > and you get the (thread.local similar) ContextVars by
> >
> > context_vars = sys.get_context_vars() # or whatever
> > context_vars.verbose = ...
>
> Migrating a (variant of a) naming subthread from python-ideas over to
> here, does the following sound plausible to anyone else?:
>
> ContextLocal - read/write access API (via get()/set() methods)
> ContextLocalNamespace - active mapping that CL.get()/set() manipulates
> ExecutionContext - current stack of context local namespaces
>
> Building a threading.local() style helper API would then look something
> like:
>
> class ContextLocals:
> def __init__(self, key_prefix):
> self._key_prefix = key_prefix
>
> def __getattr__(self, attr):
> debugging_name = "{}.{}".format(self._key_prefix, attr)
> self.__dict__[attr] = new_local =
> sys.new_context_local(debugging_name)
> return new_local
>
> def __setattr__(self, attr, value):
> getattr(self, attr).set(value)
>
> def __delattr__(self, attr):
> getattr(self, attr).set(None)
>
> my_state = ContextLocals(__name__)
>
> Cheers,
> Nick.
>
> --
> Nick Coghlan | ncogh...@gmail.com | Brisbane, Australia
> ___
> Python-Dev mailing list
> Python-Dev@python.org
> https://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe: https://mail.python.org/mailman/options/python-dev/
> guido%40python.org
>
--
--Guido van Rossum (python.org/~guido)
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe:
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Scope, not context? (was Re: PEP 550 v3 naming)
On 24 August 2017 at 15:38, Barry Warsaw wrote: > Yes, but in conversations about Python, the term “context” (in the context of > context managers) comes up way more often than the term “scope”. I actually > think Python’s scoping rules are fairly easy to grasp, as there aren’t that > many levels or ways to access them, and the natural, common interactions are > basically implicit when thinking about the code you’re writing. > > So while “context”, “environment”, and “scope” are certainly overloaded terms > in Python, the first two have very specific existing, commonplace constructs > within Python, and “scope” is both the least overloaded of the three and most > closely matches what is actually going on. > > A different tack would more closely align with PEP 550’s heritage in > thread-local storage, calling these things “execution storage”. I think I > read Guido suggest elsewhere using a namespace here so that in common code > you’d only have to change the “threading.local()” call to migrate to PEP 550. > It might be neat if you could do something like: I strongly agree with Barry's reservations about using the term "context" here. I've not been following the discussion (I was away when it started and there's too many emails to go through to catch up) but I've found the use of the term "context" to be a particular problem in trying to understand what's going on just skimming the messages. I don't have a strong opinion on what name should be used, but I am definitely against using the term "context". Paul ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Scope, not context? (was Re: PEP 550 v3 naming)
On Aug 24, 2017, at 10:23, Yury Selivanov wrote: > > On Thu, Aug 24, 2017 at 9:52 AM, Barry Warsaw wrote: >> Guido van Rossum wrote: >>> On Tue, Aug 22, 2017 at 7:12 PM, Nathaniel Smith wrote: >>> >>> I worry that that's going to lead more people astray thinking this has >>> something to do with contextlib, which it really doesn't (it's much more >>> closely related to threading.local()). >> >> This is my problem with using "Context" for this PEP. Although I can't >> keep up with all names being thrown around, it seems to me that in >> Python we already have a well-established meaning for "contexts" -- >> context managers, and the protocols they implement as they participate >> in `with` statements. We have contextlib which reinforces this. What's >> being proposed in PEP 550 is so far removed from this concept that I >> think it's just going to cause confusion (well, it does in me anyway!). > > Although nobody refers to context managers as "context", they are > always called with on of the following: "context manager", "CM", > "context manager protocol". PEP 550 just introduces a concept of > "context", that "context managers" will be able to manage. > >> >> To me, the functionality proposed in PEP 550 feels more like a "scope" >> than a "context". Unlike a lexical scope, it can't be inferred from the >> layout of the source code. It's more of a dynamic "execution scope" and >> the stacking of "local execution scopes" reinforces that for me. You >> use a key to find a value in the current execution scope, and it chains >> up until the key is found or you've reached the top of the local >> execution (defined as the thread start, etc.). > > Yes, what PEP 550 proposes can be seen as a new scoping mechanism. But > calling it a "scope" or "dynamic scope" would be a mistake (IMHO), as > Python scoping is already a complex topic (with locals, nonlocals, > globals, etc). > > Contrary to scoping, the programmer is much less likely to deal with > Execution Context. How often do we use "threading.local()”? Yes, but in conversations about Python, the term “context” (in the context of context managers) comes up way more often than the term “scope”. I actually think Python’s scoping rules are fairly easy to grasp, as there aren’t that many levels or ways to access them, and the natural, common interactions are basically implicit when thinking about the code you’re writing. So while “context”, “environment”, and “scope” are certainly overloaded terms in Python, the first two have very specific existing, commonplace constructs within Python, and “scope” is both the least overloaded of the three and most closely matches what is actually going on. A different tack would more closely align with PEP 550’s heritage in thread-local storage, calling these things “execution storage”. I think I read Guido suggest elsewhere using a namespace here so that in common code you’d only have to change the “threading.local()” call to migrate to PEP 550. It might be neat if you could do something like: import execution els = execution.local() els.x = 1 By calling it “execution local storage” you’re also providing a nicer cognitive bridge from “thread local storage”, a concept anybody diving into this stuff will already understand pretty innately. No need to get fancy, we just think “Oh, I know what thread local storage is, and this seems related but slightly different, so now I basically understand what execution local storage is”. Cheers, -Barry signature.asc Description: Message signed with OpenPGP ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Scope, not context? (was Re: PEP 550 v3 naming)
On Thu, Aug 24, 2017 at 9:52 AM, Barry Warsaw wrote: > Guido van Rossum wrote: >> On Tue, Aug 22, 2017 at 7:12 PM, Nathaniel Smith wrote: >> >> I worry that that's going to lead more people astray thinking this has >> something to do with contextlib, which it really doesn't (it's much more >> closely related to threading.local()). > > This is my problem with using "Context" for this PEP. Although I can't > keep up with all names being thrown around, it seems to me that in > Python we already have a well-established meaning for "contexts" -- > context managers, and the protocols they implement as they participate > in `with` statements. We have contextlib which reinforces this. What's > being proposed in PEP 550 is so far removed from this concept that I > think it's just going to cause confusion (well, it does in me anyway!). Although nobody refers to context managers as "context", they are always called with on of the following: "context manager", "CM", "context manager protocol". PEP 550 just introduces a concept of "context", that "context managers" will be able to manage. > > To me, the functionality proposed in PEP 550 feels more like a "scope" > than a "context". Unlike a lexical scope, it can't be inferred from the > layout of the source code. It's more of a dynamic "execution scope" and > the stacking of "local execution scopes" reinforces that for me. You > use a key to find a value in the current execution scope, and it chains > up until the key is found or you've reached the top of the local > execution (defined as the thread start, etc.). Yes, what PEP 550 proposes can be seen as a new scoping mechanism. But calling it a "scope" or "dynamic scope" would be a mistake (IMHO), as Python scoping is already a complex topic (with locals, nonlocals, globals, etc). Contrary to scoping, the programmer is much less likely to deal with Execution Context. How often do we use "threading.local()"? > One other suggestion: maybe we shouldn't put these new functions in sys, > but instead put them in their own module? It feels analogous to the gc > module; all those functions could have gone in sys since they query and > effect the Python runtime system, but it makes more sense (and improves > the naming) by putting them in their own module. It also segregates the > functionality so that sys doesn't become a catchall that overloads you > when you're reading through the sys module documentation. I'm myself not a big fan of jamming all PEP 550 APIs into the sys module. We just need to come up with a good name. Yury ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 550 leak-in vs leak-out, why not just a ChainMap
On Thu, Aug 24, 2017 at 1:12 AM, Yury Selivanov > On Thu, Aug 24, 2017 at 12:32 AM, Jim J. Jewett wrote: > The key requirement for using immutable datastructures is to make > "get_execution_context" operation fast. Do you really need the whole execution context, or do you just need the current value of a specific key? (Or, sometimes, the writable portion of the context.) > Currently, the PEP doesn't do > a good job at explaining why we need that operation and why it will be > used by asyncio.Task and call_soon, so I understand the confusion. OK, the schedulers need the whole context, but if implemented as a ChainMap (instead of per-key), isn't that just a single constant? As in, don't they always schedule to the same thread? And when they need another map, isn't that because the required context is already available from whichever code requested the scheduling? >> (A) How many values do you expect a typical generator to use? The >> django survey suggested mostly 0, sometimes 1, occasionally 2. So >> caching the values of all possible keys probably won't pay off. > Not many, but caching is still as important, because some API users > want the "get()" operation to be as fast as possible under all > conditions. Sure, but only because they view it as a hot path; if the cost of that speedup is slowing down another hot path, like scheduling the generator in the first place, it may not be worth it. According to the PEP timings, HAMT doesn't beat a copy-on-write dict until over 100 items, and never beats a regular dict.That suggests to me that it won't actually help the overall speed for a typical (as opposed to worst-case) process. >> And, of course, using a ChainMap means that the keys do NOT have to be >> predefined ... so the Key class really can be skipped. > The first version of the PEP had no ContextKey object and the most > popular complaint about it was that the key names will clash. That is true of any global registry. Thus the use of keys with prefixes like com.sun. The only thing pre-declaring a ContextKey buys in terms of clashes is that a sophisticated scheduler would have less difficulty figuring out which clashes will cause thrashing in the cache. Or are you suggesting that the key can only be declared once (as opposed to once per piece of code), so that the second framework to use the same name will see a RuntimeError? -jJ ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Scope, not context? (was Re: PEP 550 v3 naming)
On Thu, 24 Aug 2017 09:52:47 -0400 Barry Warsaw wrote: > > To me, the functionality proposed in PEP 550 feels more like a "scope" > than a "context". I would call it "environment" myself, but that risks confusion with environment variables. Perhaps "dynamic environment" would remove the confusion. > One other suggestion: maybe we shouldn't put these new functions in sys, > but instead put them in their own module? It feels analogous to the gc > module; all those functions could have gone in sys since they query and > effect the Python runtime system, but it makes more sense (and improves > the naming) by putting them in their own module. It also segregates the > functionality so that sys doesn't become a catchall that overloads you > when you're reading through the sys module documentation. +1 from me. Regards Antoine. ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
[Python-Dev] Scope, not context? (was Re: PEP 550 v3 naming)
Guido van Rossum wrote: > On Tue, Aug 22, 2017 at 7:12 PM, Nathaniel Smith wrote: > > I worry that that's going to lead more people astray thinking this has > something to do with contextlib, which it really doesn't (it's much more > closely related to threading.local()). This is my problem with using "Context" for this PEP. Although I can't keep up with all names being thrown around, it seems to me that in Python we already have a well-established meaning for "contexts" -- context managers, and the protocols they implement as they participate in `with` statements. We have contextlib which reinforces this. What's being proposed in PEP 550 is so far removed from this concept that I think it's just going to cause confusion (well, it does in me anyway!). To me, the functionality proposed in PEP 550 feels more like a "scope" than a "context". Unlike a lexical scope, it can't be inferred from the layout of the source code. It's more of a dynamic "execution scope" and the stacking of "local execution scopes" reinforces that for me. You use a key to find a value in the current execution scope, and it chains up until the key is found or you've reached the top of the local execution (defined as the thread start, etc.). One other suggestion: maybe we shouldn't put these new functions in sys, but instead put them in their own module? It feels analogous to the gc module; all those functions could have gone in sys since they query and effect the Python runtime system, but it makes more sense (and improves the naming) by putting them in their own module. It also segregates the functionality so that sys doesn't become a catchall that overloads you when you're reading through the sys module documentation. Cheers, -Barry ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 550 v3 naming
On 24 August 2017 at 08:47, Ethan Furman wrote:
>
> ContextVars is actually a different name for LogicalContext. So it would
> be:
>
> ExecutionContext = [ContextVars()[, ContextVars()[ ...]]]
>
> and you get the (thread.local similar) ContextVars by
>
> context_vars = sys.get_context_vars() # or whatever
> context_vars.verbose = ...
Migrating a (variant of a) naming subthread from python-ideas over to
here, does the following sound plausible to anyone else?:
ContextLocal - read/write access API (via get()/set() methods)
ContextLocalNamespace - active mapping that CL.get()/set() manipulates
ExecutionContext - current stack of context local namespaces
Building a threading.local() style helper API would then look something like:
class ContextLocals:
def __init__(self, key_prefix):
self._key_prefix = key_prefix
def __getattr__(self, attr):
debugging_name = "{}.{}".format(self._key_prefix, attr)
self.__dict__[attr] = new_local =
sys.new_context_local(debugging_name)
return new_local
def __setattr__(self, attr, value):
getattr(self, attr).set(value)
def __delattr__(self, attr):
getattr(self, attr).set(None)
my_state = ContextLocals(__name__)
Cheers,
Nick.
--
Nick Coghlan | ncogh...@gmail.com | Brisbane, Australia
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe:
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 550 v3 naming
On Thu, Aug 24, 2017 at 1:22 AM, Nick Coghlan wrote: > On 24 August 2017 at 02:19, Yury Selivanov wrote: >> I think that "implicit context" is not an accurate description of what >> LogicalContext is. >> >> "implicit context" only makes sense when we talk about decimal >> context. For instance, in: >> >> Decimal(1) + Decimal(2) >> >> decimal context is implicit. But this is "implicit" from the >> standpoint of that code. Decimal will manage its context >> *explicitly*. >> >> Fixing decimal context is only one part of the PEP though. EC will >> also allow to implement asynchronous task locals: >> >> current_request = new_context_key() >> >> async def handle_http_request(request): >> current_request.set(request) >> >> Here we explicitly set and will explicitly get values from the EC. We >> will explicitly manage the EC in asyncio.Task and when we schedule >> callbacks. > > The implicit behaviour that "implicit context" refers to is the fact > that if you look at an isolated snippet of code, you have no idea what > context you're actually going to be querying or modifying - that's > implicit in the execution of the whole program. How about: RuntimeContext and RuntimeContextStack -n -- Nathaniel J. Smith -- https://vorpus.org ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 550 leak-in vs leak-out, why not just a ChainMap
On Wed, Aug 23, 2017 at 9:32 PM, Jim J. Jewett wrote:
>> While the context is defined conceptually as a nested chain of
>> key:value mappings, we avoid using the mapping syntax because of the
>> way the values can shift dynamically out from under you based on who
>> called you
> ...
>> instead of having the problem of changes inside the
>> generator leaking out, we instead had the problem of
>> changes outside the generator *not* making their way in
>
> I still don't see how this is different from a ChainMap.
>
> If you are using a stack(chain) of [d_global, d_thread, d_A, d_B, d_C,
> d_mine] maps as your implicit context, then a change to d_thread map
> (that some other code could make) will be visible unless it is masked.
>
> Similarly, if the fallback for d_C changes from d_B to d_B1 (which
> points directly to d_thread), that will be visible for any keys that
> were previously resolved in d_A or d_B, or are now resolved in dB1.
>
> Those seem like exactly the cases that would (and should) cause
> "shifting values".
>
> This does mean that you can't cache everything in the localmost map,
> but that is a problem with the optimization regardless of how the
> implementation is done.
It's crucial that the only thing that can effect the result of calling
ContextKey.get() is other method calls on that same ContextKey within
the same thread. That's what enables ContextKey to efficiently cache
repeated lookups, which is an important optimization for code like
decimal or numpy that needs to access their local context *extremely*
quickly (like, a dict lookup is too slow). In fact this caching trick
is just preserving what decimal does right now in their thread-local
handling (because accessing the threadstate dict is too slow for
them).
So we can't expose any kind of mutable API to individual maps, because
then someone might call __setitem__ on some map that's lower down in
the stack, and break caching.
> And, of course, using a ChainMap means that the keys do NOT have to be
> predefined ... so the Key class really can be skipped.
The motivations for the Key class are to eliminate the need to worry
about accidental key collisions between unrelated libraries, to
provide some important optimizations (as per above), and to make it
easy and natural to provide convenient APIs like for saving and
restoring the state of a value inside a context manager. Those are all
orthogonal to whether the underlying structure is implemented as a
ChainMap or as something more specialized.
But I tend to agree with your general argument that the current PEP is
trying a bit too hard to hide away all this structure where no-one can
see it. The above constraints mean that simply exposing a ChainMap as
the public API is probably a bad idea. Even if there are compelling
performance advantages to fancy immutable-HAMT implementation (I'm in
wait-and-see mode on this myself), then there are still a lot more
introspection-y operations that could be provided, like:
- make a LocalContext out of a {ContextKey: value} dict
- get a {ContextKey: value} dict out of a LocalContext
- get the underlying list of LocalContexts out of an ExecutionContext
- create an ExecutionContext out of a list of LocalContexts
- given a ContextKey and a LocalContext, get the current value of the
key in the context
- given a ContextKey and an ExecutionContext, get out a list of values
at each level
-n
--
Nathaniel J. Smith -- https://vorpus.org
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe:
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 550 v3 naming
On 24 August 2017 at 02:19, Yury Selivanov wrote: > I think that "implicit context" is not an accurate description of what > LogicalContext is. > > "implicit context" only makes sense when we talk about decimal > context. For instance, in: > > Decimal(1) + Decimal(2) > > decimal context is implicit. But this is "implicit" from the > standpoint of that code. Decimal will manage its context > *explicitly*. > > Fixing decimal context is only one part of the PEP though. EC will > also allow to implement asynchronous task locals: > > current_request = new_context_key() > > async def handle_http_request(request): > current_request.set(request) > > Here we explicitly set and will explicitly get values from the EC. We > will explicitly manage the EC in asyncio.Task and when we schedule > callbacks. The implicit behaviour that "implicit context" refers to is the fact that if you look at an isolated snippet of code, you have no idea what context you're actually going to be querying or modifying - that's implicit in the execution of the whole program. As a user of the read/write API though, you shouldn't need to care all that much - the whole point of PEP 550 is to define default behaviours and guidelines such that it will be something sensible. Cheers, Nick. -- Nick Coghlan | ncogh...@gmail.com | Brisbane, Australia ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 550 v3 naming
On Wed, 23 Aug 2017 14:27:55 -0400 Yury Selivanov wrote: > > Yeah.. I like TaskContext when it's applied to asynchronous code. It > doesn't really work for generators because we never refer to > generators as tasks. > > Out of what was proposed so far to replace Logical Context: > > 1. DynamicContext: close to "dynamic scoping", but tries to avoid > mentioning "scopes". There are only a few languages where dynamic > scoping is implemented, so people are generally not aware of it. > > 2. ContextFrame and all frame-related names: implies that EC is > somehow implemented on top of frames or is frame-specific (which is > not always true). > > 3. ImplicitContext: covers one specific property observed from > specific examples. Context in PEP 550 is managed explicitly in some > cases. There are many use cases when API users will be working with it > explicitly too (to wrirte/read data from it). FWIW I believe that > "ExplicitContext" would be more accurate than "ImplicitContext". > > 4. LocalContext: we use "local" to describe local variables and > scoping in Python, we want to avoid any confusion here. > > 5. TaskContext: works for asynchronous tasks, but not for generators. > > I don't think that replacing LogicalContext with any name in this list > will make any improvement. I think you have a point. Though every time I want to type "logical context" it seems my fingers slip and type "local context" instead :-) Now remains the question of why the logical context stack is named "execution context" and not "logical context stack" (or "logical context chain" to keep the ChainMap analogy) :-) Regards Antoine. ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
