Re: [Python-Dev] PEP 567 -- Context Variables
On Tue, Dec 12, 2017 at 10:39 PM, Dima Tisnek wrote:
> My 2c:
> TL;DR PEP specifies implementation in some detail, but doesn't show
> how proposed change can or should be used.
>
>
>
> get()/set(value)/delete() methods: Python provides syntax sugar for
> these, let's use it.
> (dict: d["k"]/d["k] = value/del d["k"]; attrs: obj.k/obj.k = value/del
> obj.k; inheriting threading.Local)
This was already discussed to death in the PEP 550 threads... what
most users want is a single value, and routing get/set through a
ContextVar object allows for important optimizations and a simpler
implementation. Also, remember that 99% of users will never use these
objects directly; it's a low-level API mostly useful to framework
implementers.
> This PEP and 550 describe why TLS is inadequate, but don't seem to
> specify how proposed context behaves in async world. I'd be most
> interested in how it appears to work to the user of the new library.
>
> Consider a case of asynchronous cache:
>
> async def actual_lookup(name):
> ...
>
> def cached_lookup(name, cache={}):
> if name not in cache:
> cache["name"] = shield(ensure_future(actual_lookup(name))
> return cache["name"]
>
> Unrelated (or related) asynchronous processes end up waiting on the same
> future:
>
> async def called_with_user_context():
> ...
> await cached_lookup(...)
> ...
>
> Which context is propagated to actual_lookup()?
> The PEP doesn't seem to state that clearly.
> It appears to be first caller's context.
Yes.
> Is it a copy or a reference?
It's a copy, as returned by get_context().
> If first caller is cancelled, the context remains alive.
>
>
>
> token is fragile, I believe PEP should propose a working context
> manager instead.
> Btw., isn't a token really a reference to
> state-of-context-before-it's-cloned-and-modified?
No, a Token only represents the value of one ContextVar, not the whole
Context. This could maybe be clearer in the PEP, but it has to be this
way or you'd get weird behavior from code like:
with decimal.localcontext(...): # sets and then restores
numpy.seterr(...) # sets without any plan to restore
# after the 'with' block, the decimal ContextVar gets restored
# but this shouldn't affect the numpy.seterr ContextVar
-n
--
Nathaniel J. Smith -- https://vorpus.org
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe:
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 567 -- Context Variables
Hi Dima, 2017-12-13 7:39 GMT+01:00 Dima Tisnek : > get()/set(value)/delete() methods: Python provides syntax sugar for > these, let's use it. > (dict: d["k"]/d["k] = value/del d["k"]; attrs: obj.k/obj.k = value/del > obj.k; inheriting threading.Local) I was trapped by Context which is described as "a mapping". Usually, when I read "mappin", I associate it to a mutable dictionary. But in fact Context is a *read-only* mapping. Yury changed the Introduction to add "read-only", but not the Context section: https://www.python.org/dev/peps/pep-0567/#contextvars-context Only a single ContextVar variable can be modified. This object is a container for a *single* value, not a mapping, you cannot write "var = value", you have to write "var.set(value)", and "var['key] = value" doesn't make sense. > This PEP and 550 describe why TLS is inadequate, but don't seem to > specify how proposed context behaves in async world. I'd be most > interested in how it appears to work to the user of the new library. In short, context is inherited automatically, you have nothing to do :-) Put anything you want into a context, and it will follow transparently your asynchronous code. The answer is in the sentence: "Tasks in asyncio need to maintain their own context that they inherit from the point they were created at. " You may want to use a task context to pass data from a HTTP request: user name, cookie, IP address, etc. If you save data into the "current context", in practice, the context is inherited by tasks and callbacks, and so even if your code is made of multiple tasks, you still "inherit" the context as expected. Only tasks have to manually "save/restore" the context, since only tasks use "await" in their code, not callbacks called by call_soon() & cie. > token is fragile, I believe PEP should propose a working context > manager instead. Why is it fragile? In asyncio, you cannot use a context manager because of the design of tasks. Victor ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Zero-width matching in regexes
05.12.17 01:21, MRAB пише:
I've finally come to a conclusion as to what the "correct" behaviour of
zero-width matches should be: """always return the first match, but
never a zero-width match that is joined to a previous zero-width match""".
If it's about to return a zero-width match that's joined to a previous
zero-width match, then backtrack and keep on looking for a match.
Example:
>>> print([m.span() for m in re.finditer(r'|.', 'a')])
[(0, 0), (0, 1), (1, 1)]
re.findall, re.split and re.sub should work accordingly.
If re.finditer finds n matches, then re.split should return a list of
n+1 strings and re.sub should make n replacements (excepting maxsplit,
etc.).
We now have a good opportunity of changing a long standing behavior of
re.sub(). Currently empty matches are prohibited if adjacent to a
previous match. For consistency with re.finditer() and re.findall(),
with regex.sub() with VERSION1 flag, and with Perl, PCRE and other
engines they should be prohibited only if adjacent to a previous *empty*
match. Currently re.sub('x*', '-', 'abxc') returns '-a-b-c-', but will
return '-a-b--c-' if change the behavior.
This behavior already was unintentionally temporary changed between 2.1
and 2.2, when the underlying implementation of re was changed from PCRE
to SRE. But the former behavior was quickly restored (see
https://bugs.python.org/issue462270). Ironically the behavior of the
current PCRE is different.
Possible options:
1. Change the behavior right now.
2. Start emitting a FutureWarning and change the behavior in future version.
3. Keep the status quo forever.
We need to make a decision right now since in the first two cases we
should to change the behavior of re.split() right now. Its behavior is
changed in 3.7 in any case, and it is better to change the behavior once
than break the behavior in two different releases.
The changed detail is so subtle that no regular expressions in the
stdlib and tests are affected, except the special purposed test added
for guarding the current behavior.
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe:
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
[Python-Dev] PEP 489: module m_traverse called with NULL module state
Hello, After debugging a crash on AppVeyor for a submitter's PR (see https://github.com/python/cpython/pull/4611 ), I came to the following diagnosis: converting the "atexit" module (which is a built-in C extension) to PEP 489 multiphase initialization can lead to its m_traverse function (and presumably also m_clear and m_free) to be called while not module state is yet registered: that is, `PyModule_GetState(self)` when called from m_traverse returns NULL! Is that an expected or known subtlety? Regards Antoine. ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 567 -- Context Variables
Overall, I like this PEP. It's definitely easier to follow conceptually than PEP 550. Thanks for taking the time to re-think the idea. I have a few comments in-line below. -eric On Tue, Dec 12, 2017 at 10:33 AM, Yury Selivanov wrote: > This is a new proposal to implement context storage in Python. +1 This is something I've had on my back burner for years. Getting this right is non-trivial, so having a stdlib implementation will help open up clean solutions in a number of use cases that are currently addressed in more error-prone ways. > > It's a successor of PEP 550 and builds on some of its API ideas and > datastructures. Contrary to PEP 550 though, this proposal only focuses > on adding new APIs and implementing support for it in asyncio. There > are no changes to the interpreter or to the behaviour of generator or > coroutine objects. Do you have any plans to revisit extension of the concept to generators and coroutine objects? I agree they can be addressed separately, if necessary. TBH, I'd expect this PEP to provide an approach that allows such applications of the concept to effectively be implementation details that can be supported later. > Abstract > > > This PEP proposes the new ``contextvars`` module and a set of new > CPython C APIs to support context variables. This concept is > similar to thread-local variables but, unlike TLS, it allows s/it allows/it also allows/ > correctly keeping track of values per asynchronous task, e.g. > ``asyncio.Task``. > > [snip] > > Rationale > = > > Thread-local variables are insufficient for asynchronous tasks which > execute concurrently in the same OS thread. Any context manager that > needs to save and restore a context value and uses > ``threading.local()``, will have its context values bleed to other > code unexpectedly when used in async/await code. FWIW, I'd consider the concept to extend to all execution contexts in the interpreter, of which threads and async/await are the only kinds we have currently. That said, I don't see us adding any new kinds of execution context so what you've said is entirely satisfactory. :) > > [snip] > > Introduction > > > [snip] > > Specification > = > > A new standard library module ``contextvars`` is added Why not add this to contextlib instead of adding a new module? IIRC this was discussed relative to PEP 550, but I don't remember the reason. Regardless, it would be worth mentioning somewhere in the PEP. > with the > following APIs: > > 1. ``get_context() -> Context`` function is used to get the current >``Context`` object for the current OS thread. > > 2. ``ContextVar`` class to declare and access context variables. It may be worth explaining somewhere in the PEP the reason why you've chosen to add ContextVar instead of adding a new keyword (e.g. "context", a la global and nonlocal) to do roughly the same thing. Consider that execution contexts are very much a language-level concept, a close sibling to scope. Driving that via a keyword would a reasonable approach, particularly since it introduces less coupling between a language-level feature and a stdlib module. (Making it a builtin would sort of help with that too, but a keyword would seem like a better fit.) A keyword would obviate the need for explicitly calling .get() and .set(). FWIW, I agree with not adding a new keyword. To me context variables are a low-level tool for library authors to implement their high-level APIs. ContextVar, with its explicit .get() and .set() methods is a good fit for that and better communicates the conceptual intent of the feature. However, it would still be worth explicitly mentioning the alternate keyword-based approach in the PEP. > > 3. ``Context`` class encapsulates context state. Every OS thread >stores a reference to its current ``Context`` instance. >It is not possible to control that reference manually. >Instead, the ``Context.run(callable, *args)`` method is used to run >Python code in another context. I'd call that "Context.call()" since its for callables. Did you have a specific reason for calling it "run" instead? > > FWIW, I think there are some helpers you could add that library authors would appreciate. However, they aren't critical so I'll hold off and maybe post about them later. :) > contextvars.ContextVar > -- > > The ``ContextVar`` class has the following constructor signature: > ``ContextVar(name, *, default=no_default)``. The ``name`` parameter > is used only for introspection and debug purposes. It doesn't need to be required then, right? > [snip] > > ``ContextVar.set(value) -> Token`` is used to set a new value for > the context variable in the current ``Context``:: > > # Set the variable 'var' to 1 in the current context. > var.set(1) > > ``contextvars.Token`` is an opaque object that should be used to > restore the ``ContextVar`` to its previous value, or remove it from > the
Re: [Python-Dev] PEP 567 -- Context Variables
On Tue, Dec 12, 2017 at 4:49 PM, Victor Stinner wrote: >> The ``Token`` API exists to make the current proposal forward >> compatible with :pep:`550`, in case there is demand to support >> context variables in generators and asynchronous generators in the >> future. > > Cool. I like the idea of starting with something simple in Python 3.7. > Then extend it in Python 3.8 or later (support generators), if it > becomes popular, once the first simple (but "incomplete", without > generators) implementation is battle-tested. +1 for starting with a basic API and building on that. -eric ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 567 -- Context Variables
Hi Eric, Thanks for a detailed review! On Wed, Dec 13, 2017 at 3:59 PM, Eric Snow wrote: > Overall, I like this PEP. It's definitely easier to follow > conceptually than PEP 550. Thanks for taking the time to re-think the > idea. I have a few comments in-line below. > > -eric > > On Tue, Dec 12, 2017 at 10:33 AM, Yury Selivanov > wrote: >> This is a new proposal to implement context storage in Python. > > +1 > > This is something I've had on my back burner for years. Getting this > right is non-trivial, so having a stdlib implementation will help open > up clean solutions in a number of use cases that are currently > addressed in more error-prone ways. Right! > >> >> It's a successor of PEP 550 and builds on some of its API ideas and >> datastructures. Contrary to PEP 550 though, this proposal only focuses >> on adding new APIs and implementing support for it in asyncio. There >> are no changes to the interpreter or to the behaviour of generator or >> coroutine objects. > > Do you have any plans to revisit extension of the concept to > generators and coroutine objects? I agree they can be addressed > separately, if necessary. TBH, I'd expect this PEP to provide an > approach that allows such applications of the concept to effectively > be implementation details that can be supported later. Maybe we'll extend the concept to work for generators in Python 3.8, but that's a pretty remote topic to discuss (and we'll need a new PEP for that). In case we decide to do that, PEP 550 provides a good implementation plan, and PEP 567 are forward-compatible with it. > >> Abstract >> >> >> This PEP proposes the new ``contextvars`` module and a set of new >> CPython C APIs to support context variables. This concept is >> similar to thread-local variables but, unlike TLS, it allows > > s/it allows/it also allows/ Will fix it. [..] >> A new standard library module ``contextvars`` is added > > Why not add this to contextlib instead of adding a new module? IIRC > this was discussed relative to PEP 550, but I don't remember the > reason. Regardless, it would be worth mentioning somewhere in the > PEP. > The mechanism is generic and isn't directly related to context managers. Context managers can (and in many cases should) use the new APIs to store global state, but the contextvars APIs do not depend on context managers or require them. I also feel that contextlib is a big module already, so having the new APIs in their separate module and having a separate documentation page makes it more approachable. >> with the >> following APIs: >> >> 1. ``get_context() -> Context`` function is used to get the current >>``Context`` object for the current OS thread. >> >> 2. ``ContextVar`` class to declare and access context variables. > > It may be worth explaining somewhere in the PEP the reason why you've > chosen to add ContextVar instead of adding a new keyword (e.g. > "context", a la global and nonlocal) to do roughly the same thing. > Consider that execution contexts are very much a language-level > concept, a close sibling to scope. Driving that via a keyword would a > reasonable approach, particularly since it introduces less coupling > between a language-level feature and a stdlib module. (Making it a > builtin would sort of help with that too, but a keyword would seem > like a better fit.) A keyword would obviate the need for explicitly > calling .get() and .set(). > > FWIW, I agree with not adding a new keyword. To me context variables > are a low-level tool for library authors to implement their high-level > APIs. ContextVar, with its explicit .get() and .set() methods is a > good fit for that and better communicates the conceptual intent of the > feature. However, it would still be worth explicitly mentioning the > alternate keyword-based approach in the PEP. Yeah, adding keywords is way harder than adding a new module. It would require a change in Grammar, new opcodes, changes to frameobject etc. I also don't think that ContextVars will be that popular to have their own syntax -- how many threadlocals do you see every day? For PEP 567/550 a keyword isn't really needed, we can implement the concept with a ContextVar class. >> >> 3. ``Context`` class encapsulates context state. Every OS thread >>stores a reference to its current ``Context`` instance. >>It is not possible to control that reference manually. >>Instead, the ``Context.run(callable, *args)`` method is used to run >>Python code in another context. > > I'd call that "Context.call()" since its for callables. Did you have > a specific reason for calling it "run" instead? We have a bunch of run() methods in asyncio, and as I'm actively working on its codebase I might be biased here, but ".run()" reads better for me personally than ".call()". > FWIW, I think there are some helpers you could add that library > authors would appreciate. However, they aren't critical so I'll hold > off and maybe post abo
Re: [Python-Dev] Support of the Android platform
Hi Xavier, I looked at your scripts to build Android but I failed to use them. Anyway, I'm not sure why these scripts have to be part of the CPython git repository. Technically, is there a reason to put it aside the source code and Unix build scripts (configure/Makefile/setup)? Your https://github.com/python/cpython/pull/1629 only adds new files without touching existing files. I suggest to create new Git project. It may be in the python organization, or you may start with your GitHub account. Cross-compilation is hard, and I'm not sure that it's possible to build a single recipe for all Android API versions, all configuration, any set of libraries, etc. For Android, it seems like each developer might want a subtle different configuration which might not be easy to support. Having a separated Git project would allow people to contribute more easily, experiment their fork, etc. What do you think? I'm only talking about the proposed Android/ directory and https://github.com/python/cpython/pull/1629. Everything else is fine :-) Victor 2017-12-10 15:19 GMT+01:00 Xavier de Gaye : > The following note is a proposal to add the support of the Android platform. > > The note is easier to read with clickable links at > https://github.com/xdegaye/cagibi/blob/master/doc/android_support.rst > > Motivations > === > > * Android is ubiquitous. > * This would be the first platform supported by Python that is > cross-compiled, > thanks to many contributors. > * Although the Android operating system is linux, it is different from most > linux platforms, for example it does not use GNU libc and runs SELinux in > enforcing mode. Therefore supporting this platform would make Python more > robust and also would allow testing it on arm 64-bit processors. > * Python running on Android is also a handheld calculator, a successor of > the > slide rule and the `HP 41`_. > > Current status > == > > * The Python test suite succeeds when run on Android emulators using > buildbot > strenuous settings with the following architectures on API 24: x86, > x86_64, > armv7 and arm64. > * The `Android build system`_ is described in another section. > * The `buildmaster-config PR 26`_ proposes to update ``master.cfg`` to > enable > buildbots to run a given Android API and architecture on the emulators. > * The Android emulator is actually ``qemu``, so the test suites for x86 and > x86_64 last about the same time as the test suite run natively when the > processor of the build system is of the x86 family. The test suites for > the > arm architectures last much longer: about 8 hours for arm64 and 10 hours > for > armv7 on a four years old laptop. > * The changes that have been made to achieve this status are listed in > `bpo-26865`_, the Android meta-issue. > * Given the cpu resources required to run the test suite on the arm > emulators, > it may be difficult to find a contributed buildbot worker. So it remains > to > find the hardware to run these buildbots. > > Proposal > > > Support the Android platform on API 24 [1]_ for the x86_64, armv7 and arm64 > architectures built with NDK 14b. > > *API 24* > * API 21 is the first version to provide usable support for wide > characters > and where SELinux is run in enforcing mode. > > * API 22 introduces an annoying bug on the linker that prints something > like > this when python is started:: > > ``WARNING: linker: libpython3.6m.so.1.0: unused DT entry: type > 0x6ffe arg 0x14554``. > > The `termux`_ Android terminal emulator describes this problem at the > end > of its `termux-packages`_ gitlab page and has implemented a > ``termux-elf-cleaner`` tool to strip the useless entries from the ELF > header of executables. > > * API 24 is the first version where the `adb`_ shell is run on the > emulator > as a ``shell`` user instead of the ``root`` user previously, and the > first > version that supports arm64. > > *x86_64* > It seems that no handheld device exists using that architecture. It is > supported because the x86_64 Android emulator runs fast and therefore is a > good candidate as a buildbot worker. > > *NDK 14b* > This release of the NDK is the first one to use `Unified headers`_ fixing > numerous problems that had been fixed by updating the Python configure > script > until now (those changes have been reverted by now). > > Android idiosyncrasies > == > > * The default shell is ``/system/bin/sh``. > * The file system layout is not a traditional unix layout, there is no > ``/tmp`` for example. Most directories have user restricted access, > ``/sdcard`` is mounted as ``noexec`` for example. > * The (java) applications are allocated a unix user id and a subdirectory on > ``/data/data``. > * SELinux is run in enforcing mode. > * Shared memory and semaphores are not supported. > * The default encoding is UTF-8. > > Android build system > ===
Re: [Python-Dev] PEP 540: Add a new UTF-8 mode (v3)
On 11 Dec. 2017 6:50 am, "INADA Naoki" wrote: Except one typo I commented on Github, I accept PEP 540. Well done, Victor and Nick for PEP 540 and 538. Python 3.7 will be most UTF-8 friendly Python 3 than ever. And thank you for all of your work on reviewing them! The appropriate trade-offs between ease of use in common scenarios and an increased chance of emitting mojibake are hard to figure out, but I like where we've ended up :) Cheers, Nick. ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 565: Show DeprecationWarning in __main__
On 13 Dec. 2017 12:53 pm, "Victor Stinner" wrote: 2017-12-13 0:24 GMT+01:00 Guido van Rossum : > Considered disagreement is acceptable. Sure, I'm fine with that ;-) > Nick, congrats with PEP 565! Please update the PEP to mark it as approved > with a link to this message as the resolution, and let's get the > implementation into 3.7a4! Nick wrote that he will be away, since I update his PEP: https://github.com/python/peps/commit/355eced94cf4117492c9e1eee8f950 f08e53ec90 Thanks Guido for the approval, and Victor for explaining the dev mode connection updating the PEP status! I'll get the implementation updated & merged in the first week of January (my phone is my only client device for most of the time until then). Cheers, Nick. ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 489: module m_traverse called with NULL module state
On 14 Dec. 2017 9:19 am, "Antoine Pitrou" wrote: Hello, After debugging a crash on AppVeyor for a submitter's PR (see https://github.com/python/cpython/pull/4611 ), I came to the following diagnosis: converting the "atexit" module (which is a built-in C extension) to PEP 489 multiphase initialization can lead to its m_traverse function (and presumably also m_clear and m_free) to be called while not module state is yet registered: that is, `PyModule_GetState(self)` when called from m_traverse returns NULL! Is that an expected or known subtlety? Not that I'm aware of, so I'd be inclined to classify it as a bug in the way we're handling multi-phase initialisation unless/until we determine there's no way to preserve the existing invariant from the single phase case. Cheers, Nick. ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
