Re: [Python-Dev] PEP 514: Python registration in the Windows registry
On 16Jul2016 1254, Paul Moore wrote: On 15 July 2016 at 23:20, Steve Dower wrote: Hi all I'd like to get this PEP approved (status changed to Active, IIUC). Some comments below. Awesome, thanks! Posted a pull request at https://github.com/python/peps/pull/59 for ease of diff reading, and some commentary below (with aggressive snipping). Motivation == When installed on Windows, the official Python installer creates a registry key for discovery and detection by other applications. This allows tools such as installers or IDEs to automatically detect and display a user's Python installations. The PEP seems quite strongly focused on GUI tools ... I'd like to avoid tool developers reading this section and thinking "it only applies to GUI tools or OS integration, not to me". Agreed. I tried to avoid any console/GUI-specific terms, but I can probably be more explicit about it being useful to both. For example, virtualenv introspects the available Python installations - see https://github.com/pypa/virtualenv/blob/master/virtualenv.py#L86 - to support the "-p " flag. To handle this well, it would be useful to allow distributions to register a "short tag", so that as well as "-p 3.5" or "-p 2", Virtualenv could support (say) "-p conda3.4" or "-p pypy2". (The short tag should be at the Company level, so "conda" or "pypy", and the version gets added to that). Another place where this might be useful is the py.exe launcher (it's not in scope for this PEP, but having the data needed to allow the launcher to invoke any available installation could be useful for future enhancements). virtualenv would be a great example to use. My thinking was that the Tag should be appropriate here (perhaps with the Company to disambiguate when necessary), and that is now explicit. Anaconda currently has "Anaconda_4.1.1_64-bit" as their tag, which would not be convenient, so an explicit suggestion here would help ensure this is useful. Another key motivation for me would be to define clearly what information tools can rely on being able to get from the available registry entries describing what's installed. Whenever I've needed to scan the registry, the things I've needed to find out are where I find the Python interpreter, what Python version it is, and whether it's 32-bit or 64-bit. The first so that I can run Python, and the latter two so that I can tell if this is a version I support *without* needing to run the interpreter. For me, everything else in this PEP is about UI, but those 3 items plus the "short tag" idea are more about what capabilities I can provide. Good points. I discussed architecture with a colleague at one point and I'm not entirely sure it's universally useful (what architecture is IronPython when built for Any CPU? what architecture is Jython?), but maybe something like the contents of importlib.machinery.IMPORT_SUFFIXES would be? On 64-bit Windows, ``HKEY_LOCAL_MACHINE\Software\Wow6432Node`` is a special key that 32-bit processes transparently read and write to rather than accessing the ``Software`` key directly. It might be worth being more explicit here that 32-bit and 64-bit processes see the registry keys slightly differently. More on this below. I considered this and thought I had a link to the official docs about it. I don't want this PEP to be mistaken for documentation on how registry redirection works :) Backwards Compatibility --- Also, Python 3.5 doesn't appear to include the architecture in sys.winver either. ... (Unless it adds -32 for 32-bit, and reserves the bare version for 64-bit. I've skimmed the CPython source but can't confirm that). The documentation of sys.winver makes no mention of whether it distinguishes 32- and 64-bit builds. In fact, it states "The value is normally the first three characters of version". If we're relying on sys.winver being unique by version/architecture, the docs need to say so (so that future changes don't accidentally violate that). I'll update the docs, but your guess is correct. I changed sys.winver on 32-bit to be "3.5-32" since that matches what py.exe was already using to refer to it. I didn't want to invent yet-another-way-to-tag-architectures. (I also updated py.exe to match tags directly under PythonCore, so 3.5-32 is matched without scanning the binary type.) Also, sys.winver is defined in PCBuild/python.props, which is how we accidentally backported the suffix to 2.7.11 :( It is not possible to detect side-by-side installations of both 64-bit and 32-bit versions of Python prior to 3.5 when they have been installed for the current user. Python 3.5 and later always uses different Tags for 64-bit and 32-bit versions. From what I can see, this latter isn't true. I presume that 64-bit uses no suffix, but 32-bit uses a "-32" suffix? This should probably be made explicit. At a minimum, if I were writing a tool to list all installed Python versions, with only what I h
Re: [Python-Dev] PEP 514: Python registration in the Windows registry
On 18 July 2016 at 17:33, Steve Dower wrote: >> Some comments below. > > Awesome, thanks! Posted a pull request at > https://github.com/python/peps/pull/59 for ease of diff reading, and some > commentary below (with aggressive snipping). Thanks - I'll do a proper review of that, but just wanted to make a few comments here. > virtualenv would be a great example to use. My thinking was that the Tag > should be appropriate here (perhaps with the Company to disambiguate when > necessary), and that is now explicit. > > Anaconda currently has "Anaconda_4.1.1_64-bit" as their tag, which would not > be convenient, so an explicit suggestion here would help ensure this is > useful. Yeah, that's not a useful value for this use case. What I'm thinking of is that currently a number of projects (for example, virtualenv, tox, and a personal Powershell wrapper I have round virtualenv) do this registry introspection exercise, purely to provide a "more convenient" way of specifying a Python version than giving the full path to the interpreter. Unix users have versioned executables, so -p python3.5 works fine, but Windows users don't have that. So my idea is "something as easy to remember as python3.5". But having said this, we're talking about a theoretical extension to existing functionality, that probably has marginal utility at best, so I don't want to get hung up on details here. > Snipped most of the details because I agree it's unsatisfying right now, but > I disagree with enough of the counterproposal that it was getting to be > messy commenting on each bit. I take your points here. What I was trying to avoid (because I've encountered it myself) is having to actually *run* the Python interpreter to extract this information. Unix code does this freely, because running subprocesses is so cheap there, but starting up a load of processes on Windows is a non-trivial cost. But again, this is in the area of "potential use cases" rather than "we need it now", so I'm OK with deferring the question if you're uncertain. OK, that's enough off-the-cuff responses. I'll find some time to review your PR (probably tomorrow) and comment there. Paul ___ Python-Dev mailing list [email protected] https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP487: Simpler customization of class creation
Great job, Martin! Thanks for seeing this through. :) -eric On Sun, Jul 17, 2016 at 10:57 AM, Guido van Rossum wrote: > This PEP is now accepted for inclusion in Python 3.6. Martin, > congratulations! Someone (not me) needs to review and commit your > changes, before September 12, when the 3.6 feature freeze goes into > effect (see https://www.python.org/dev/peps/pep-0494/#schedule). > > On Sun, Jul 17, 2016 at 4:32 AM, Martin Teichmann > wrote: >> Hi Guido, Hi Nick, Hi list, >> >> so I just updated PEP 487, you can find it here: >> https://github.com/python/peps/pull/57 if it hasn't already been >> merged. There are no substantial updates there, I only updated the >> wording as suggested, and added some words about backwards >> compatibility as hinted by Nick. >> >> Greetings >> >> Martin >> >> 2016-07-14 17:47 GMT+02:00 Guido van Rossum : >>> I just reviewed the changes you made, I like __set_name__(). I'll just >>> wait for your next update, incorporating Nick's suggestions. Regarding >>> who merges PRs to the PEPs repo, since you are the author the people >>> who merge don't pass any judgment on the changes (unless it doesn't >>> build cleanly or maybe if they see a typo). If you intend a PR as a >>> base for discussion you can add a comment saying e.g. "Don't merge >>> yet". If you call out @gvanrossum, GitHub will make sure I get a >>> message about it. >>> >>> I think the substantial discussion about the PEP should remain here in >>> python-dev; comments about typos, grammar and other minor editorial >>> issues can go on GitHub. Hope this part of the process makes sense! >>> >>> On Thu, Jul 14, 2016 at 6:50 AM, Martin Teichmann >>> wrote: Hi Guido, Hi list, Thanks for the nice review! I applied followed up your ideas and put it into a github pull request: https://github.com/python/peps/pull/53 Soon we'll be working there, until then, some responses to your comments: > I wonder if this should be renamed to __set_name__ or something else > that clarifies we're passing it the name of the attribute? The method > name __set_owner__ made me assume this is about the owning object > (which is often a useful term in other discussions about objects), > whereas it is really about telling the descriptor the name of the > attribute for which it applies. The name for this has been discussed a bit already, __set_owner__ was Nick's idea, and indeed, the owner is also set. Technically, __set_owner_and_name__ would be correct, but actually I like your idea of __set_name__. > That (inheriting type from type, and object from object) is very > confusing. Why not just define new classes e.g. NewType and NewObject > here, since it's just pseudo code anyway? Actually, it's real code. If you drop those lines at the beginning of the tests for the implementation (as I have done here: https://github.com/tecki/cpython/blob/pep487b/Lib/test/test_subclassinit.py), the test runs on older Pythons. But I see that my idea to formulate things here in Python was a bad idea, I will put the explanation first and turn the code into pseudo-code. >> def __init__(self, name, bases, ns, **kwargs): >> super().__init__(name, bases, ns) > > What does this definition of __init__ add? It removes the keyword arguments. I describe that in prose a bit down. >> class object: >> @classmethod >> def __init_subclass__(cls): >> pass >> >> class object(object, metaclass=type): >> pass > > Eek! Too many things named object. Well, I had to do that to make the tests run... I'll take that out. >> In the new code, it is not ``__init__`` that complains about keyword >> arguments, >> but ``__init_subclass__``, whose default implementation takes no >> arguments. In >> a classical inheritance scheme using the method resolution order, each >> ``__init_subclass__`` may take out it's keyword arguments until none are >> left, >> which is checked by the default implementation of ``__init_subclass__``. > > I called this out previously, and I am still a bit uncomfortable with > the backwards incompatibility here. But I believe what you describe > here is the compromise proposed by Nick, and if that's the case I have > peace with it. No, this is not Nick's compromise, this is my original. Nick just sent another mail to this list where he goes a bit more into the details, I'll respond to that about this topic. Greetings Martin P.S.: I just realized that my changes to the PEP were accepted by someone else than Guido. I am a bit surprised about that, but I guess this is how it works? ___ Python-Dev mailing list >
Re: [Python-Dev] PEP 467: Minor API improvements to bytes, bytearray, and memoryview
On 06/07/2016 02:34 PM, Koos Zevenhoven wrote: Why not bytes.viewbytes (or whatever name) so that one could also subscript it? And if it were a property, one could perhaps conveniently get the n'th byte: b'abcde'.viewbytes[n] # compared to b'abcde'[n:n+1] AFAICT, 'viewbytes' doesn't add much over bytes itself if we add a 'getbyte' method. Also, would it not be more clear to call the int -> bytes method something like bytes.fromint or bytes.fromord and introduce the same thing on str? And perhaps allow multiple arguments to create a str/bytes of length > 1. I guess this may violate TOOWTDI, but anyway, just a thought. Yes, it would. Changing to 'bytes.fromint'. -- ~Ethan~ ___ Python-Dev mailing list [email protected] https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
[Python-Dev] PEP 467: next round
Taking into consideration the comments from the last round:
- 'bytes.zeros' renamed to 'bytes.size', with option byte filler
(defaults to b'\x00')
- 'bytes.byte' renamed to 'fromint', add 'bchr' function
- deprecation and removal softened to deprecation/discouragement
---
PEP: 467
Title: Minor API improvements for binary sequences
Version: $Revision$
Last-Modified: $Date$
Author: Nick Coghlan , Ethan Furman
Status: Draft
Type: Standards Track
Content-Type: text/x-rst
Created: 2014-03-30
Python-Version: 3.6
Post-History: 2014-03-30 2014-08-15 2014-08-16 2016-06-07
Abstract
During the initial development of the Python 3 language specification, the
core ``bytes`` type for arbitrary binary data started as the mutable type
that is now referred to as ``bytearray``. Other aspects of operating in
the binary domain in Python have also evolved over the course of the Python
3 series.
This PEP proposes five small adjustments to the APIs of the ``bytes``,
``bytearray`` and ``memoryview`` types to make it easier to operate entirely
in the binary domain:
* Deprecate passing single integer values to ``bytes`` and ``bytearray``
* Add ``bytes.size`` and ``bytearray.size`` alternative constructors
* Add ``bytes.fromint`` and ``bytearray.fromint`` alternative constructors
* Add ``bytes.getbyte`` and ``bytearray.getbyte`` byte retrieval methods
* Add ``bytes.iterbytes``, ``bytearray.iterbytes`` and
``memoryview.iterbytes`` alternative iterators
Proposals
=
Deprecation of current "zero-initialised sequence" behaviour without removal
Currently, the ``bytes`` and ``bytearray`` constructors accept an integer
argument and interpret it as meaning to create a zero-initialised sequence
of the given size::
>>> bytes(3)
b'\x00\x00\x00'
>>> bytearray(3)
bytearray(b'\x00\x00\x00')
This PEP proposes to deprecate that behaviour in Python 3.6, but to leave
it in place for at least as long as Python 2.7 is supported, possibly
indefinitely.
No other changes are proposed to the existing constructors.
Addition of explicit "count and byte initialised sequence" constructors
---
To replace the deprecated behaviour, this PEP proposes the addition of an
explicit ``size`` alternative constructor as a class method on both
``bytes`` and ``bytearray`` whose first argument is the count, and whose
second argument is the fill byte to use (defaults to ``\x00``)::
>>> bytes.size(3)
b'\x00\x00\x00'
>>> bytearray.size(3)
bytearray(b'\x00\x00\x00')
>>> bytes.size(5, b'\x0a')
b'\x0a\x0a\x0a\x0a\x0a'
>>> bytearray.size(5, b'\x0a')
bytearray(b'\x0a\x0a\x0a\x0a\x0a')
It will behave just as the current constructors behave when passed a single
integer.
Addition of "bchr" function and explicit "single byte" constructors
---
As binary counterparts to the text ``chr`` function, this PEP proposes
the addition of a ``bchr`` function and an explicit ``fromint`` alternative
constructor as a class method on both ``bytes`` and ``bytearray``::
>>> bchr(ord("A"))
b'A'
>>> bchr(ord(b"A"))
b'A'
>>> bytes.fromint(65)
b'A'
>>> bytearray.fromint(65)
bytearray(b'A')
These methods will only accept integers in the range 0 to 255 (inclusive)::
>>> bytes.fromint(512)
Traceback (most recent call last):
File "", line 1, in
ValueError: integer must be in range(0, 256)
>>> bytes.fromint(1.0)
Traceback (most recent call last):
File "", line 1, in
TypeError: 'float' object cannot be interpreted as an integer
The documentation of the ``ord`` builtin will be updated to explicitly note
that ``bchr`` is the primary inverse operation for binary data, while ``chr``
is the inverse operation for text data, and that ``bytes.fromint`` and
``bytearray.fromint`` also exist.
Behaviourally, ``bytes.fromint(x)`` will be equivalent to the current
``bytes([x])`` (and similarly for ``bytearray``). The new spelling is
expected to be easier to discover and easier to read (especially when used
in conjunction with indexing operations on binary sequence types).
As a separate method, the new spelling will also work better with higher
order functions like ``map``.
Addition of "getbyte" method to retrieve a single byte
--
This PEP proposes that ``bytes`` and ``bytearray`` gain the method ``getbyte``
which will always return ``bytes``::
>>> b'abc'.getbyte(0)
b'a'
If an index is asked for that doesn't exist, ``IndexError`` is raised::
>>> b'abc'.getbyte(9)
Traceback (most recent call last):
File "", line 1, in
IndexError: index out of range
Addition of optimised iterator methods that produce ``bytes`` objects
---
Re: [Python-Dev] PEP 467: next round
On Mon, Jul 18, 2016 at 4:17 PM, Ethan Furman wrote: > - 'bytes.zeros' renamed to 'bytes.size', with option byte filler > (defaults to b'\x00') > Seriously? You went from a numpy-friendly feature to something rather numpy-hostile. In numpy, ndarray.size is an attribute that returns the number of elements in the array. The constructor that creates an arbitrary repeated value also exists and is called numpy.full(). Even ignoring numpy, bytes.size(count, value=b'\x00') is completely unintuitive. If I see bytes.size(42) in someone's code, I will think: "something like int.bit_length(), but in bytes." ___ Python-Dev mailing list [email protected] https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 467: next round
*de-lurks* On Mon, Jul 18, 2016 at 4:45 PM, Alexander Belopolsky wrote: > On Mon, Jul 18, 2016 at 4:17 PM, Ethan Furman wrote: >> >> - 'bytes.zeros' renamed to 'bytes.size', with option byte filler >> (defaults to b'\x00') > > > Seriously? You went from a numpy-friendly feature to something rather > numpy-hostile. > In numpy, ndarray.size is an attribute that returns the number of elements > in the array. > > The constructor that creates an arbitrary repeated value also exists and is > called numpy.full(). > > Even ignoring numpy, bytes.size(count, value=b'\x00') is completely > unintuitive. If I see bytes.size(42) in someone's code, I will think: > "something like int.bit_length(), but in bytes." full(), despite its use in numpy, is also unintuitive to me (my first thought is that it would indicate whether an object has room for more entries). Perhaps bytes.fillsize? That would seem the most intuitive to me: "fill an object of this size with this byte". I'm unfamiliar with numpy, but a quick Google search suggests that this would not conflict with anything there, if that is a concern. > This PEP isn't revisiting that original design decision, just changing the > spelling as users sometimes find the current behaviour of the binary > sequence > constructors surprising. In particular, there's a reasonable case to be made > that ``bytes(x)`` (where ``x`` is an integer) should behave like the > ``bytes.byte(x)`` proposal in this PEP. Providing both behaviours as > separate > class methods avoids that ambiguity. You have a leftover bytes.byte here. ___ Python-Dev mailing list [email protected] https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 467: next round
On Mon, Jul 18, 2016 at 5:01 PM, Jonathan Goble wrote: > full(), despite its use in numpy, is also unintuitive to me (my first > thought is that it would indicate whether an object has room for more > entries). > > Perhaps bytes.fillsize? > I wouldn't want to see bytes.full() either. Maybe bytes.of_size()? ___ Python-Dev mailing list [email protected] https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 467: next round
On Mon, 18 Jul 2016 at 14:35 Alexander Belopolsky < [email protected]> wrote: > > On Mon, Jul 18, 2016 at 5:01 PM, Jonathan Goble > wrote: > >> full(), despite its use in numpy, is also unintuitive to me (my first >> thought is that it would indicate whether an object has room for more >> entries). >> >> Perhaps bytes.fillsize? >> > > I wouldn't want to see bytes.full() either. Maybe bytes.of_size()? > Or bytes.fromsize() to stay with the trend of naming constructor methods as from*() ? ___ Python-Dev mailing list [email protected] https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 467: next round
On 07/18/2016 02:01 PM, Jonathan Goble wrote: This PEP isn't revisiting that original design decision, just changing the spelling as users sometimes find the current behaviour of the binary sequence constructors surprising. In particular, there's a reasonable case to be made that ``bytes(x)`` (where ``x`` is an integer) should behave like the ``bytes.byte(x)`` proposal in this PEP. Providing both behaviours as separate class methods avoids that ambiguity. You have a leftover bytes.byte here. Thanks, fixed (plus the other couple locations ;) -- ~Ethan~ ___ Python-Dev mailing list [email protected] https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 467: next round
On 07/18/2016 02:45 PM, Brett Cannon wrote: On Mon, 18 Jul 2016 at 14:35 Alexander Belopolsky wrote: On Mon, Jul 18, 2016 at 5:01 PM, Jonathan Goble wrote: full(), despite its use in numpy, is also unintuitive to me (my first thought is that it would indicate whether an object has room for more entries). Perhaps bytes.fillsize? I wouldn't want to see bytes.full() either. Maybe bytes.of_size()? Or bytes.fromsize() to stay with the trend of naming constructor methods as from*() ? bytes.fromsize() sounds good to me, thanks for brainstorming that one for me. I wasn't really happy with 'size()' either. -- ~Ethan~ ___ Python-Dev mailing list [email protected] https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 467: next round
On Mon, Jul 18, 2016 at 6:00 PM, Ethan Furman wrote: > bytes.fromsize() sounds good to me, thanks for brainstorming that one for > me. > +1 ___ Python-Dev mailing list [email protected] https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 467: next round
On Mon, Jul 18, 2016, at 17:34, Alexander Belopolsky wrote: > On Mon, Jul 18, 2016 at 5:01 PM, Jonathan Goble > wrote: > > > full(), despite its use in numpy, is also unintuitive to me (my first > > thought is that it would indicate whether an object has room for more > > entries). > > > > Perhaps bytes.fillsize? > > I wouldn't want to see bytes.full() either. Maybe bytes.of_size()? What's wrong with b'\0'*42? ___ Python-Dev mailing list [email protected] https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 467: next round
On Mon, 18 Jul 2016 at 15:49 Random832 wrote: > On Mon, Jul 18, 2016, at 17:34, Alexander Belopolsky wrote: > > On Mon, Jul 18, 2016 at 5:01 PM, Jonathan Goble > > wrote: > > > > > full(), despite its use in numpy, is also unintuitive to me (my first > > > thought is that it would indicate whether an object has room for more > > > entries). > > > > > > Perhaps bytes.fillsize? > > > > I wouldn't want to see bytes.full() either. Maybe bytes.of_size()? > > What's wrong with b'\0'*42? > It's mentioned in the PEP as to why. ___ Python-Dev mailing list [email protected] https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP487: Simpler customization of class creation
Yes, I'm very excited about this! Will this mean no more metaclass conflicts if using @abstractmethod? On Sun, Jul 17, 2016 at 12:59 PM Guido van Rossum wrote: > This PEP is now accepted for inclusion in Python 3.6. Martin, > congratulations! Someone (not me) needs to review and commit your > changes, before September 12, when the 3.6 feature freeze goes into > effect (see https://www.python.org/dev/peps/pep-0494/#schedule). > > On Sun, Jul 17, 2016 at 4:32 AM, Martin Teichmann > wrote: > > Hi Guido, Hi Nick, Hi list, > > > > so I just updated PEP 487, you can find it here: > > https://github.com/python/peps/pull/57 if it hasn't already been > > merged. There are no substantial updates there, I only updated the > > wording as suggested, and added some words about backwards > > compatibility as hinted by Nick. > > > > Greetings > > > > Martin > > > > 2016-07-14 17:47 GMT+02:00 Guido van Rossum : > >> I just reviewed the changes you made, I like __set_name__(). I'll just > >> wait for your next update, incorporating Nick's suggestions. Regarding > >> who merges PRs to the PEPs repo, since you are the author the people > >> who merge don't pass any judgment on the changes (unless it doesn't > >> build cleanly or maybe if they see a typo). If you intend a PR as a > >> base for discussion you can add a comment saying e.g. "Don't merge > >> yet". If you call out @gvanrossum, GitHub will make sure I get a > >> message about it. > >> > >> I think the substantial discussion about the PEP should remain here in > >> python-dev; comments about typos, grammar and other minor editorial > >> issues can go on GitHub. Hope this part of the process makes sense! > >> > >> On Thu, Jul 14, 2016 at 6:50 AM, Martin Teichmann > >> wrote: > >>> Hi Guido, Hi list, > >>> > >>> Thanks for the nice review! I applied followed up your ideas and put > >>> it into a github pull request: https://github.com/python/peps/pull/53 > >>> > >>> Soon we'll be working there, until then, some responses to your > comments: > >>> > I wonder if this should be renamed to __set_name__ or something else > that clarifies we're passing it the name of the attribute? The method > name __set_owner__ made me assume this is about the owning object > (which is often a useful term in other discussions about objects), > whereas it is really about telling the descriptor the name of the > attribute for which it applies. > >>> > >>> The name for this has been discussed a bit already, __set_owner__ was > >>> Nick's idea, and indeed, the owner is also set. Technically, > >>> __set_owner_and_name__ would be correct, but actually I like your idea > >>> of __set_name__. > >>> > That (inheriting type from type, and object from object) is very > confusing. Why not just define new classes e.g. NewType and NewObject > here, since it's just pseudo code anyway? > >>> > >>> Actually, it's real code. If you drop those lines at the beginning of > >>> the tests for the implementation (as I have done here: > >>> > https://github.com/tecki/cpython/blob/pep487b/Lib/test/test_subclassinit.py > ), > >>> the test runs on older Pythons. > >>> > >>> But I see that my idea to formulate things here in Python was a bad > >>> idea, I will put the explanation first and turn the code into > >>> pseudo-code. > >>> > > def __init__(self, name, bases, ns, **kwargs): > > super().__init__(name, bases, ns) > > What does this definition of __init__ add? > >>> > >>> It removes the keyword arguments. I describe that in prose a bit down. > >>> > > class object: > > @classmethod > > def __init_subclass__(cls): > > pass > > > > class object(object, metaclass=type): > > pass > > Eek! Too many things named object. > >>> > >>> Well, I had to do that to make the tests run... I'll take that out. > >>> > > In the new code, it is not ``__init__`` that complains about keyword > arguments, > > but ``__init_subclass__``, whose default implementation takes no > arguments. In > > a classical inheritance scheme using the method resolution order, > each > > ``__init_subclass__`` may take out it's keyword arguments until none > are left, > > which is checked by the default implementation of > ``__init_subclass__``. > > I called this out previously, and I am still a bit uncomfortable with > the backwards incompatibility here. But I believe what you describe > here is the compromise proposed by Nick, and if that's the case I have > peace with it. > >>> > >>> No, this is not Nick's compromise, this is my original. Nick just sent > >>> another mail to this list where he goes a bit more into the details, > >>> I'll respond to that about this topic. > >>> > >>> Greetings > >>> > >>> Martin > >>> > >>> P.S.: I just realized that my changes to the PEP were accepted by > >>> someone else than Guido. I am a bit surprised
[Python-Dev] Fun with ExitStack
I was trying to debug a problem in some work code and I ran into some
interesting oddities with contextlib.ExitStack and other context managers in
Python 3.5.
This program creates a temporary directory, and I wanted to give it a --keep
flag to not automatically delete the tempdir at program exit. I was using an
ExitStack to manage a bunch of resources, and the temporary directory is the
first thing pushed into the ExitStack. At that point in the program, I check
the value of --keep and if it's set, I use ExitStack.pop_all() to clear the
stack, and thus, presumably, prevent the temporary directory from being
deleted.
There's this relevant quote in the contextlib documentation:
"""
Each instance [of an ExitStack] maintains a stack of registered callbacks that
are called in reverse order when the instance is closed (either explicitly or
implicitly at the end of a with statement). Note that callbacks are not
invoked implicitly when the context stack instance is garbage collected.
"""
However if I didn't save the reference to the pop_all'd ExitStack, the tempdir
would be immediately deleted. If I did save a reference to the pop_all'd
ExitStack, the tempdir would live until the saved reference went out of scope
and got refcounted away.
As best I can tell this happens because TemporaryDirectory.__init__() creates
a weakref finalizer which ends up calling the _cleanup() function. Although
it's rather difficult to trace, it does appear that when the ExitStack is
gc'd, this finalizer gets triggered (via weakref.detach()), thus causing the
_cleanup() method to be called and the tmpdir to get deleted. I "fix" this by
doing:
def __init__(self):
tmpdir = TemporaryDirectory()
self._tmpdir = (tmpdir.name if keep
else self.resources.enter_context(tmpdir))
There must be more to the story because when __init__() exits in the --keep
case, tmpdir should have gotten refcounted away and the directory deleted, but
it doesn't. I haven't dug down deep enough to figure that out.
Now, while I was debugging that behavior, I ran across more interesting bits.
I put this in a file to drive some tests:
--snip snip-
with ExitStack() as resources:
print('enter context')
tmpdir = resources.enter_context(X())
resources.pop_all()
print('exit context')
--snip snip-
Let's say X is:
class X:
def __enter__(self):
print('enter Foo')
return self
def __exit__(self, *args, **kws):
print('exit Foo')
return False
the output is:
enter context
enter Foo
exit context
So far so good. A fairly standard context manager class doesn't get its
__exit__() called even when the program exits. Let's try this:
@contextmanager
def X():
print('enter bar')
yield
print('exit bar')
still good:
enter context
enter bar
exit context
Let's modify X a little bit to be a more common idiom:
@contextmanager
def X():
print('enter foo')
try:
yield
finally:
print('exit foo')
enter context
enter foo
exit foo
exit context
Ah, the try-finally changes the behavior! There's probably some documentation
somewhere that defines how a generator gets finalized, and that triggers the
finally clause, whereas in the previous example, nothing after the yield gets
run. I just can't find that anything that would describe the observed
behavior.
It's all very twisty, and I'm not sure Python is doing anything wrong, but I'm
also not sure it's *not* doing anything wrong. ;)
In any case, the contextlib documentation quoted above should probably be more
liberally sprinkled with salty caveats. Just calling .pop_all() isn't
necessarily enough to ensure that resources managed by an ExitStack will
survive its garbage collection.
Cheers,
-Barry
pgpZ3dUkiDZhm.pgp
Description: OpenPGP digital signature
___
Python-Dev mailing list
[email protected]
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe:
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Fun with ExitStack
On 18 July 2016 at 23:40, Barry Warsaw wrote:
> I was trying to debug a problem in some work code and I ran into some
> interesting oddities with contextlib.ExitStack and other context managers in
> Python 3.5.
>
> This program creates a temporary directory, and I wanted to give it a --keep
> flag to not automatically delete the tempdir at program exit. I was using an
> ExitStack to manage a bunch of resources, and the temporary directory is the
> first thing pushed into the ExitStack. At that point in the program, I check
> the value of --keep and if it's set, I use ExitStack.pop_all() to clear the
> stack, and thus, presumably, prevent the temporary directory from being
> deleted.
>
> There's this relevant quote in the contextlib documentation:
>
> """
> Each instance [of an ExitStack] maintains a stack of registered callbacks that
> are called in reverse order when the instance is closed (either explicitly or
> implicitly at the end of a with statement). Note that callbacks are not
> invoked implicitly when the context stack instance is garbage collected.
> """
>
> However if I didn't save the reference to the pop_all'd ExitStack, the tempdir
> would be immediately deleted. If I did save a reference to the pop_all'd
> ExitStack, the tempdir would live until the saved reference went out of scope
> and got refcounted away.
>
> As best I can tell this happens because TemporaryDirectory.__init__() creates
> a weakref finalizer which ends up calling the _cleanup() function. Although
> it's rather difficult to trace, it does appear that when the ExitStack is
> gc'd, this finalizer gets triggered (via weakref.detach()), thus causing the
> _cleanup() method to be called and the tmpdir to get deleted.
This seems to be missing from the documentation, but when you delete a
TemporaryDirectory instance without using a context manager or
cleanup(), it complains, and then cleans up anyway:
>>> d = TemporaryDirectory()
>>> del d
/usr/lib/python3.5/tempfile.py:797: ResourceWarning: Implicitly
cleaning up
_warnings.warn(warn_message, ResourceWarning)
Perhaps you will have to use the lower-level mkdtemp() function
instead if you want the option of making the “temporary” directory
long-lived.
> I "fix" this by
> doing:
>
> def __init__(self):
> tmpdir = TemporaryDirectory()
> self._tmpdir = (tmpdir.name if keep
> else self.resources.enter_context(tmpdir))
>
> There must be more to the story because when __init__() exits in the --keep
> case, tmpdir should have gotten refcounted away and the directory deleted, but
> it doesn't. I haven't dug down deep enough to figure that out.
>
> Now, while I was debugging that behavior, I ran across more interesting bits.
> I put this in a file to drive some tests:
>
> --snip snip-
> with ExitStack() as resources:
> print('enter context')
> tmpdir = resources.enter_context(X())
> resources.pop_all()
> print('exit context')
> --snip snip-
>
> Let's say X is:
>
> class X:
> def __enter__(self):
> print('enter Foo')
> return self
>
> def __exit__(self, *args, **kws):
> print('exit Foo')
> return False
>
> the output is:
>
> enter context
> enter Foo
> exit context
>
> So far so good. A fairly standard context manager class doesn't get its
> __exit__() called even when the program exits. Let's try this:
>
> @contextmanager
> def X():
> print('enter bar')
> yield
> print('exit bar')
>
> still good:
>
> enter context
> enter bar
> exit context
>
> Let's modify X a little bit to be a more common idiom:
>
> @contextmanager
> def X():
> print('enter foo')
> try:
> yield
> finally:
> print('exit foo')
>
> enter context
> enter foo
> exit foo
> exit context
>
> Ah, the try-finally changes the behavior! There's probably some documentation
> somewhere that defines how a generator gets finalized, and that triggers the
> finally clause, whereas in the previous example, nothing after the yield gets
> run. I just can't find that anything that would describe the observed
> behavior.
I suspect the documentation doesn’t spell everything out, but my
understanding is that garbage collection of a generator instance
effectively calls its close() method, triggering any “finally” and
__exit__() handlers.
IMO, in some cases if a generator would execute these handlers and
gets garbage collected, it is a programming error because the
programmer should have explicitly called generator.close(). In these
cases, it would be nice to emit a ResourceWarning, just like
forgetting to close a file, or delete your temporay directory above.
But maybe there are other cases where there is no valid reason to emit
a warning. I have hesitated in suggesting this change in the past, but
I don’t remember why. One reason is it might be an annoyance with code
that also wants to handle non-generator iterators that don’t have a
close() method. In a world before “yiel
Re: [Python-Dev] PEP 467: next round
(Thanks for moving this forward, Ethan!)
On 19 July 2016 at 06:17, Ethan Furman wrote:
> * Add ``bytes.getbyte`` and ``bytearray.getbyte`` byte retrieval methods
> * Add ``bytes.iterbytes``, ``bytearray.iterbytes`` and
> ``memoryview.iterbytes`` alternative iterators
As a possible alternative to this aspect, what if we adjusted
memorview.cast() to also support the "s" format code from the struct
module?
At the moment, trying to use "s" gives a value error:
>>> bview = memoryview(data).cast("s")
Traceback (most recent call last):
File "", line 1, in
ValueError: memoryview: destination format must be a native single
character format prefixed with an optional '@'
However, it could be supported by always interpreting it as equivalent
to "1s", such that the view produced length 1 bytes objects on
indexing and iteration, rather than integers (which is what it does
given the default "b" format).
Given "memoryview(data).cast('s')" as a basic building block, most of
the other aspects of working with bytes objects as if they were Python
2 strings should become relatively straightforward, so the question
would be whether we wanted to make it easy for people to avoid
constructing the mediating memoryview object.
> Proposals
> =
>
> Deprecation of current "zero-initialised sequence" behaviour without removal
>
>
> Currently, the ``bytes`` and ``bytearray`` constructors accept an integer
> argument and interpret it as meaning to create a zero-initialised sequence
> of the given size::
>
> >>> bytes(3)
> b'\x00\x00\x00'
> >>> bytearray(3)
> bytearray(b'\x00\x00\x00')
>
> This PEP proposes to deprecate that behaviour in Python 3.6, but to leave
> it in place for at least as long as Python 2.7 is supported, possibly
> indefinitely.
I'd suggest being more explicit that this would just be a documented
deprecation, rather than a programmatic deprecatation warning.
> Addition of explicit "count and byte initialised sequence" constructors
> ---
>
> To replace the deprecated behaviour, this PEP proposes the addition of an
> explicit ``size`` alternative constructor as a class method on both
> ``bytes`` and ``bytearray`` whose first argument is the count, and whose
> second argument is the fill byte to use (defaults to ``\x00``)::
>
> >>> bytes.size(3)
> b'\x00\x00\x00'
> >>> bytearray.size(3)
> bytearray(b'\x00\x00\x00')
> >>> bytes.size(5, b'\x0a')
> b'\x0a\x0a\x0a\x0a\x0a'
> >>> bytearray.size(5, b'\x0a')
> bytearray(b'\x0a\x0a\x0a\x0a\x0a')
While I like the notion of having "size" in the name, the
"noun-as-constructor" phrasing doesn't read right to me. Perhaps
"fromsize" for consistency with "fromhex"?
> It will behave just as the current constructors behave when passed a single
> integer.
This last paragraph feels incomplete now, given the expansion to allow
the fill value to be specified.
> Addition of "bchr" function and explicit "single byte" constructors
> ---
>
> As binary counterparts to the text ``chr`` function, this PEP proposes
> the addition of a ``bchr`` function and an explicit ``fromint`` alternative
> constructor as a class method on both ``bytes`` and ``bytearray``::
>
> >>> bchr(ord("A"))
> b'A'
> >>> bchr(ord(b"A"))
> b'A'
> >>> bytes.fromint(65)
> b'A'
> >>> bytearray.fromint(65)
> bytearray(b'A')
Since "fromsize" would also accept an int value, "fromint" feels
ambiguous here. Perhaps "fromord" to emphasise the integer is being
interpreted as an ordinal bytes value, rather than as a size?
The apparent "two ways to do it" here also deserves some additional explanation:
- the bchr builtin is to recreate the ord/chr/unichr trio from Python
2 under a different naming scheme
- the class method is mainly for the "bytearray.fromord" case, with
bytes.fromord added for consistency
[snip sections on accessing elements as bytes object]
> Design discussion
> =
>
> Why not rely on sequence repetition to create zero-initialised sequences?
> -
>
> Zero-initialised sequences can be created via sequence repetition::
>
> >>> b'\x00' * 3
> b'\x00\x00\x00'
> >>> bytearray(b'\x00') * 3
> bytearray(b'\x00\x00\x00')
>
> However, this was also the case when the ``bytearray`` type was originally
> designed, and the decision was made to add explicit support for it in the
> type constructor. The immutable ``bytes`` type then inherited that feature
> when it was introduced in PEP 3137.
>
> This PEP isn't revisiting that original design decision, just changing the
> spelling as users sometimes find the current behaviour of the binary
> sequence
> constructors surprising. In particular, there's a reasonable ca
Re: [Python-Dev] PEP 467: next round
On 19 July 2016 at 08:00, Ethan Furman wrote: > On 07/18/2016 02:45 PM, Brett Cannon wrote: >> >> On Mon, 18 Jul 2016 at 14:35 Alexander Belopolsky wrote: >>> >>> On Mon, Jul 18, 2016 at 5:01 PM, Jonathan Goble wrote: > > full(), despite its use in numpy, is also unintuitive to me (my first thought is that it would indicate whether an object has room for more entries). Perhaps bytes.fillsize? >>> >>> >>> I wouldn't want to see bytes.full() either. Maybe bytes.of_size()? >> >> >> Or bytes.fromsize() to stay with the trend of naming constructor methods >> as from*() ? > > > bytes.fromsize() sounds good to me, thanks for brainstorming that one for > me. I wasn't really happy with 'size()' either. Heh, I should have finished reading the thread before replying - this and one of my other comments were already picked up :) Cheers, Nick. -- Nick Coghlan | [email protected] | Brisbane, Australia ___ Python-Dev mailing list [email protected] https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP487: Simpler customization of class creation
On 19 July 2016 at 09:26, Neil Girdhar wrote: > Yes, I'm very excited about this! > > Will this mean no more metaclass conflicts if using @abstractmethod? ABCMeta and EnumMeta both create persistent behavioural differences rather than only influencing subtype definition, so they'll need to remain as custom metaclasses. What this PEP (especially in combination with PEP 520) is aimed at enabling is subclassing APIs designed more around the notion of "implicit class decoration" where a common base class or mixin can be adjusted to perform certain actions whenever a new subclass is defined, without changing the runtime behaviour of those subclasses. (For example: a mixin or base class may require that certain parameters be set as class attributes - this PEP will allow the base class to check for those and throw an error at definition time, rather than getting a potentially cryptic error when it attempts to use the missing attribute) Cheers, Nick. -- Nick Coghlan | [email protected] | Brisbane, Australia ___ Python-Dev mailing list [email protected] https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Fun with ExitStack
On 19 July 2016 at 09:40, Barry Warsaw wrote: > Ah, the try-finally changes the behavior! There's probably some documentation > somewhere that defines how a generator gets finalized, and that triggers the > finally clause, whereas in the previous example, nothing after the yield gets > run. I just can't find that anything that would describe the observed > behavior. For the generator case, their __del__ calls self.close(), and that throws a GeneratorExit exception into the current yield point. Since it's an exception, that will run try/finally clauses and context manager __exit__ methods, but otherwise bypass code after the yield statement. > It's all very twisty, and I'm not sure Python is doing anything wrong, but I'm > also not sure it's *not* doing anything wrong. ;) I think we're a bit inconsistent in how we treat the lazy cleanup for managed resources - sometimes __del__ handles it, sometimes we register a weakref finalizer, sometimes we don't do it at all. That makes it hard to predict precise behaviour without knowing the semantic details of the specific context managers involved in the stack. > In any case, the contextlib documentation quoted above should probably be more > liberally sprinkled with salty caveats. Just calling .pop_all() isn't > necessarily enough to ensure that resources managed by an ExitStack will > survive its garbage collection. Aye, I'd be open to changes that clarified that even though the ExitStack instance won't invoke any cleanup callbacks explicitly following a pop_all(), *implicit* cleanup from its references to those objects going away may still be triggered if you don't save the result of the pop_all() call somewhere. Cheers, Nick. -- Nick Coghlan | [email protected] | Brisbane, Australia ___ Python-Dev mailing list [email protected] https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP487: Simpler customization of class creation
Yes, I see what you're saying. However, I don't understand why __init_subclass__ (defined on some class C) cannot be used to implement the checks required by @abstractmethod instead of doing it in ABCMeta. This would prevent metaclass conflicts since you could use @abstractmethod with any metaclass or no metaclass at all provided you inherit from C. On Tue, Jul 19, 2016 at 12:21 AM Nick Coghlan wrote: > On 19 July 2016 at 09:26, Neil Girdhar wrote: > > Yes, I'm very excited about this! > > > > Will this mean no more metaclass conflicts if using @abstractmethod? > > ABCMeta and EnumMeta both create persistent behavioural differences > rather than only influencing subtype definition, so they'll need to > remain as custom metaclasses. > > What this PEP (especially in combination with PEP 520) is aimed at > enabling is subclassing APIs designed more around the notion of > "implicit class decoration" where a common base class or mixin can be > adjusted to perform certain actions whenever a new subclass is > defined, without changing the runtime behaviour of those subclasses. > (For example: a mixin or base class may require that certain > parameters be set as class attributes - this PEP will allow the base > class to check for those and throw an error at definition time, rather > than getting a potentially cryptic error when it attempts to use the > missing attribute) > > Cheers, > Nick. > > -- > Nick Coghlan | [email protected] | Brisbane, Australia > ___ Python-Dev mailing list [email protected] https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
