Re: [Python-Dev] Guarantee the success of some object creation C API functions
On 02.05.17 00:52, Chris Angelico wrote: Aside from straight-up bugs, how can one of these functions fail? Is memory allocation failure the only way? Yes, memory allocation failure is the only way. In normal Python build this can happen only at early stage of the interpreter initialization, until the first call of the function allocate a singleton in dynamic memory, or, for some instances, at late stage of the interpreter finalization after deallocating singletons. In custom Python build with disabled free lists for small ints, tuples, etc (if define NSMALLPOSINTS or PyTuple_MAXSAVESIZE to 0) this can happen at any time. Incidentally, this guarantee, if implemented the obvious way, will also mean that (), "", 0, etc are singletons. People talk casually about the "empty tuple singleton", but I don't think it's actually guaranteed anywhere. As noted Nick this guarantee doesn't go so far. With C API you can modify just created tuple in-place and set its size to 0. This gives you an empty tuple different from the "empty tuple singleton". Some C API functions actually can return non-singletons equal to singleton objects, but usually we try to avoid this. Not because there are any guaranties, but for optimization. References to singletons consume less memory and compare faster. ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] __getattribute__'s error is not available in __getattr__
On 2 May 2017 at 13:47, Jason Maldonis wrote: > Hi everyone, > > If this should be asked in learn python I apologize -- please just tell me > without answering. > > I'm working on a large class architecture and I find myself often > overloading __getattr__. I am continuously running into the issue where I > want __getattr__ to have access to the error that was raised in > __getattribute__, but it seems completely unavailable. Is that true? __getattr__ can be called *from* __getattribute__, so when it runs, __getattribute__ hasn't necessarily failed yet - it may just be on its last resort strategy for attribute retrieval. If you're sure the base class __getattribute__ doesn't call __getattr__ directly, you can do: def __getattribute__(self, name): try: return super().__getattribute__(name) except AttributeError: return self.__getattr__(name) However, would you mind filing a documentation bug for this? I can't find anything in the language or library reference that explicitly states whether or not `object.__getattribute__` itself calls `__getattr__` directly, and that's a docs limitation which should be addressed. Cheers, Nick. -- Nick Coghlan | ncogh...@gmail.com | Brisbane, Australia ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Guarantee the success of some object creation C API functions
On 2 May 2017 at 07:52, Chris Angelico wrote: > On Tue, May 2, 2017 at 6:52 AM, Terry Reedy wrote: >> The promise makes it clear that breaking the property is a bug to be fixed. >> It only decreases the probability for someone who has read the promise. >> Unfortunately, 'never fail' is hard to test ;-). >> > > Aside from straight-up bugs, how can one of these functions fail? Is > memory allocation failure the only way? If so, the proposed > implementation (private references to pre-created singletons) ought to > guarantee that, to the exact extent that anything else can be > guaranteed. > > (Or is that your point - that "never fail" is always "modulo bugs"?) > > Incidentally, this guarantee, if implemented the obvious way, will > also mean that (), "", 0, etc are singletons. People talk casually > about the "empty tuple singleton", but I don't think it's actually > guaranteed anywhere. I don't think it is either, and I don't think it's a guarantee we want to make - even with Serhiy's proposed change, it's still straightforward to produce non-singleton instances of these values using the low level C APIs, while the true singletons (True, False, None, Ellipsis) go out of their way to make it difficult to create new instances of the corresponding types, even when mucking about at the C layer. The assertion Serhiy is proposing to make is much weaker, and would be a matter of adding something like the following to the C API reference: "Similar to the singleton values True, False, None, and Ellipsis, process global instances of the builtin values (), '', b'', 0, and 1 are pre-allocated at interpreter startup, so APIs returning these values should never fail, even in low memory conditions. However, additional empty instances of these types may still be created through the C API, so they should always be compared by value rather than by identity." The specific "never fails as it returns a pointer to a pre-allocated instance" cases can then be documented on the affected public API functions. So +1 from me for making pre-allocation a CPython C API guarantee, -1 for making it a language level singleton commitment. Cheers, Nick. -- Nick Coghlan | ncogh...@gmail.com | Brisbane, Australia ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
[Python-Dev] __getattribute__'s error is not available in __getattr__
Hi everyone, If this should be asked in learn python I apologize -- please just tell me without answering. I'm working on a large class architecture and I find myself often overloading __getattr__. I am continuously running into the issue where I want __getattr__ to have access to the error that was raised in __getattribute__, but it seems completely unavailable. Is that true? One simple case that I'm guessing others have run into, is if __getattr__ fails, the error from __getattribute__ isn't in the stack trace that gets printed to screen. To fix this (on occasion) I'll even re-call __getattribute__ within __getattr__ just to get the error so I can properly "raise from" the __getattibute__'s error -- although that's probably bad practice in general. I'd like to be able to access the error that was raised in __getattribute__ when __getattr__ is called. Two more quick context comments: python is awesome, thank you all for your hard work; and I've been writing python almost every day for ~ 5 years now and I can do all the "black magic" jazz, so I'll be okay with an implementation that requires that type of stuff if necessary. Thanks! Jason ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Guarantee the success of some object creation C API functions
On Tue, May 2, 2017 at 6:52 AM, Terry Reedy wrote: > The promise makes it clear that breaking the property is a bug to be fixed. > It only decreases the probability for someone who has read the promise. > Unfortunately, 'never fail' is hard to test ;-). > Aside from straight-up bugs, how can one of these functions fail? Is memory allocation failure the only way? If so, the proposed implementation (private references to pre-created singletons) ought to guarantee that, to the exact extent that anything else can be guaranteed. (Or is that your point - that "never fail" is always "modulo bugs"?) Incidentally, this guarantee, if implemented the obvious way, will also mean that (), "", 0, etc are singletons. People talk casually about the "empty tuple singleton", but I don't think it's actually guaranteed anywhere. ChrisA ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Guarantee the success of some object creation C API functions
On 5/1/2017 5:50 AM, Serhiy Storchaka wrote:
I want to add promises to public C API functions that create trivial
instances of immutable basic types (integers 0 and 1, empty tuple,
string and bytes object) -- PyLong_FromLong(0), PyLong_FromLong(1),
PyTuple_New(0), PyUnicode_FromStringAndSize(NULL, 0),
PyUnicode_FromString(""), PyBytes_FromStringAndSize(NULL, 0),
PyBytes_FromString("") -- that they always succeed and never return
NULL. Currently they always or almost always succeed, but this is an
implementation detail. I want to make this promise official and more
strong, so that it is true even at very early stage of the interpreter
intitalization and at very late stage of the finalization. Also I want
to add private references to singletons that represent these values
(_PyLong_Zero, _PyTuple_Empty, etc) similar to Py_None and Py_True.
This could make some code clearer. For example see [1] and [2].
These promises add limitations for future changes of the interpreter,
but the official promise decreases the probability of unintentional
changes that break implicit properties.
The promise makes it clear that breaking the property is a bug to be
fixed. It only decreases the probability for someone who has read the
promise. Unfortunately, 'never fail' is hard to test ;-).
Do I have the right to give such promises?
By yourself, I would think not. With a couple of coredev concurrences,
and none against, maybe. But I think the following does need to be
asked publicly, here, so implementors of CPython alternatives have a
chance to respond.
> Are there any reasons that might prevent their implementation?
If the answer is no, I think the simplifications in the following are a
good argument for the change. On the other hand, I have to wonder why
there were not made previously.
[1] https://bugs.python.org/issue29878
[2] https://bugs.python.org/issue30162
--
Terry Jan Reedy
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe:
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Request review of cProfile/profile series issue
For who may be reviewing cProfile / profile context manager problem, I prepare a post for explaining why the problem become, and why the patch will fix this problem: post: https://blog.louie.lu/2017/04/23/python-libs-profile-cant-context-manager/ Thanks, Louie. 2017-04-23 11:21 GMT+08:00 Louie Lu : > Hi all, > > I'm now looking for cProfile/profile lib's issue, and have solve a series of > dependent problem, here is the list: > > #9285 - Add a profile decorator to profile and cProfile > #30113 - Allow helper functions to wrap sys.setprofile > #18971 - Use argparse in the profile/cProfile modules > #30118 - Add unittest for cProfile/profile command line interface > > It can divide into two categories, first is the context manager problem, and > the second is optparse to argparse problem. > > 1. context manager problem: > > Relative issue: #9285, #30113 > Relative PR: #287, #1212, #1253 > > This is an issue since 2010, and stop at profile can't simply add a > __enter__ and __exit__ to make it a context manager. The main problem is, > sys.setprofile() will hit the return and get bad return in profile > dispatch_return function. The solution is to insert a simulate call in the > helper function, to provide the context between helper frame and where the > profile is defined. > > 2. optparse to argparse problem: > > Relative issue: #18971, #30118 > Relative PR: #1232 > Relative patch: profile_argparse.patch > > Serhiy have provide the patch of replace optparse to argparse in the profile > and cProfile, but block by Ezio request of unittest for command line > interface. The unittest is provide at #1232, and need to be reivew. If the > unittest is add and argparse patch is apply, we can then solve more problem, > e.g.: #23420, #29238, #21862 > > > This is what I've investigated for cProfile / profile library now, > to be move on, it will need others to review the work. > > Thanks! > > Best Regards, > Louie. ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Escaping docs markup in NEWS entries?
Where would be a good place for test cases for an rst_escape() function? Docutils? https://github.com/westurner/dotfiles/blob/develop/scripts/git-changelog.py - rst_escape # YMMV - $ git-changelog.py -r "release/0.3.14" --hdr= "+"` On Monday, May 1, 2017, Nick Coghlan wrote: > On 1 May 2017 at 17:13, Martin Panter > > wrote: > > On 1 May 2017 at 06:37, Nick Coghlan > > wrote: > >> Hi folks, > >> > >> I'm trying to write a NEWS entry that explains that the > >> ":func:`bytes`" cross-references have changed to refer to the type > >> descriptions by default (matching other builtin container types), so > >> you now need to use ``:ref:`func-bytes`" to refer to the old target in > >> the list of builtin functions (if you really want that for some > >> reason). > >> > >> Unfortunately, my first two attempts both cause warnings in "make > >> suspicious" with the following output: > > > > What is the full output? Usually it includes instructions to add false > > positives to Doc/tools/susp-ignored.csv; maybe that is all you have to > > do? > > You're right, that would be likely be the way to go if I decided to > keep the escaped markup. > > However... > > >> My fallback plan is to just include the unescaped markup in the NEWS > >> entry (rather than trying to make it readable even in rendered form), > >> but I figured I'd ask for advice here first. > > > > I thought the NEWS file was mainly regarded as plain text, so it would > > be important to avoid ugly RST markup like backslashes if possible. > > ... I think you're right on this point, so it makes more sense to skip > the escaping entirely, > and just use the correct link markup in the NEWS entry. How convenient. > > Cheers, > Nick. > > -- > Nick Coghlan | ncogh...@gmail.com| Brisbane, > Australia > ___ > Python-Dev mailing list > Python-Dev@python.org > https://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: https://mail.python.org/mailman/options/python-dev/ > wes.turner%40gmail.com > ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
[Python-Dev] Guarantee the success of some object creation C API functions
I want to add promises to public C API functions that create trivial
instances of immutable basic types (integers 0 and 1, empty tuple,
string and bytes object) -- PyLong_FromLong(0), PyLong_FromLong(1),
PyTuple_New(0), PyUnicode_FromStringAndSize(NULL, 0),
PyUnicode_FromString(""), PyBytes_FromStringAndSize(NULL, 0),
PyBytes_FromString("") -- that they always succeed and never return
NULL. Currently they always or almost always succeed, but this is an
implementation detail. I want to make this promise official and more
strong, so that it is true even at very early stage of the interpreter
intitalization and at very late stage of the finalization. Also I want
to add private references to singletons that represent these values
(_PyLong_Zero, _PyTuple_Empty, etc) similar to Py_None and Py_True.
This could make some code clearer. For example see [1] and [2].
These promises add limitations for future changes of the interpreter,
but the official promise decreases the probability of unintentional
changes that break implicit properties.
Do I have the right to give such promises? Are there any reasons that
might prevent their implementation?
[1] https://bugs.python.org/issue29878
[2] https://bugs.python.org/issue30162
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe:
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Escaping docs markup in NEWS entries?
On 1 May 2017 at 17:13, Martin Panter wrote: > On 1 May 2017 at 06:37, Nick Coghlan wrote: >> Hi folks, >> >> I'm trying to write a NEWS entry that explains that the >> ":func:`bytes`" cross-references have changed to refer to the type >> descriptions by default (matching other builtin container types), so >> you now need to use ``:ref:`func-bytes`" to refer to the old target in >> the list of builtin functions (if you really want that for some >> reason). >> >> Unfortunately, my first two attempts both cause warnings in "make >> suspicious" with the following output: > > What is the full output? Usually it includes instructions to add false > positives to Doc/tools/susp-ignored.csv; maybe that is all you have to > do? You're right, that would be likely be the way to go if I decided to keep the escaped markup. However... >> My fallback plan is to just include the unescaped markup in the NEWS >> entry (rather than trying to make it readable even in rendered form), >> but I figured I'd ask for advice here first. > > I thought the NEWS file was mainly regarded as plain text, so it would > be important to avoid ugly RST markup like backslashes if possible. ... I think you're right on this point, so it makes more sense to skip the escaping entirely, and just use the correct link markup in the NEWS entry. Cheers, Nick. -- Nick Coghlan | ncogh...@gmail.com | Brisbane, Australia ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Escaping docs markup in NEWS entries?
On 1 May 2017 at 06:37, Nick Coghlan wrote: > Hi folks, > > I'm trying to write a NEWS entry that explains that the > ":func:`bytes`" cross-references have changed to refer to the type > descriptions by default (matching other builtin container types), so > you now need to use ``:ref:`func-bytes`" to refer to the old target in > the list of builtin functions (if you really want that for some > reason). > > Unfortunately, my first two attempts both cause warnings in "make > suspicious" with the following output: What is the full output? Usually it includes instructions to add false positives to Doc/tools/susp-ignored.csv; maybe that is all you have to do? > > WARNING: [whatsnew/changelog:986] ":func" found in "\:func\:\`bytes\`" > WARNING: [whatsnew/changelog:986] "`" found in "\:func\:\`bytes\`" > WARNING: [whatsnew/changelog:986] ":func" found in "\:func\:\`bytearray\`" > WARNING: [whatsnew/changelog:986] "`" found in "\:func\:\`bytearray\`" > WARNING: [whatsnew/changelog:986] ":ref" found in "\:ref\:\`func-bytes\`" > WARNING: [whatsnew/changelog:986] "`" found in "\:ref\:\`func-bytes\`" > WARNING: [whatsnew/changelog:986] ":ref" found in "\:ref\:\`func-bytes\`" > WARNING: [whatsnew/changelog:986] "`" found in "\:ref\:\`func-bytes\`" > > > My first attempt just escaped the nested backticks: > > - Issue #30052: the link targets for ``:func:\`bytes\``` and > ``:func:\`bytearray\``` are now their respective type definitions, rather > than the corresponding builtin function entries. Use ``:ref:\`func-bytes\``` > and ``:ref:\`func-bytes\``` to reference the latter. > > My second attempt escaped the colons as well: > > - Issue #30052: the link targets for ``\:func\:\`bytes\``` and > ``\:func\:\`bytearray\``` are now their respective type definitions, rather > than the corresponding builtin function entries. Use > ``\:ref\:\`func-bytes\``` > and ``\:ref\:\`func-bytes\``` to reference the latter. > > My fallback plan is to just include the unescaped markup in the NEWS > entry (rather than trying to make it readable even in rendered form), > but I figured I'd ask for advice here first. I thought the NEWS file was mainly regarded as plain text, so it would be important to avoid ugly RST markup like backslashes if possible. I am no RST expert, but perhaps it would be clearer to a human RST parser if you added a space among the last three backticks, where the underscore is in ``:func:`bytes`_``. ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
