[Python-Dev] slice subscripts for sequences and mappings
Hello, I find a strange discrepancy in Python with regards to slice subscripting of objects, at the C API level. I mean things like obj[start:end:step]. I'd expect slice subscripts to be part of the sequence interface, and yet they are not. In fact, they are part of the mapping interface. For example, the list object has its slice get/set methods assigned to a PyMappingMethods struct. So does a bytes object, and pretty much every other object that wants to support subscripts. This doesn't align well with the documentation, in at least two places. 1) The library documentation (http://docs.python.org/dev/library/stdtypes.html) in 4.8 says: Mappings are mutable objects. There is currently only one standard mapping type, the dictionary Why then does a list implement the mapping interface? Moreover, why does bytes, an immutable object, implement the mapping interface? 2) The same documentation page in 4.6 says, in the operation table: s[i:j] slice of s from i to j s[i:j:k] slice of s from i to j with step k But in the implementation, the slice subscripts are part of the mapping, not the sequence inteface. The PySequenceMethods structure does have fields for slice accessors, but their naming (was_sq_slice, was_sq_ass_slice) suggests they're just deprecated placeholders. This also doesn't align well with logic, since mappings like dict have no real meaning for slice subscripts. These logically belong to a sequence. Moreover, it takes subscripts for single a single numeric index away from subscripts for a slice into a different protocol (the former in sequence, the latter in mapping). I realize I must be missing some piece of the history here and not suggesting to change anything. I do think that the documentation, especially in the area of the type object that defines the sequence and mapping protocols, could be clarified to express what is expected of a new type that wants to act as a sequence. In particular, it should be said explicitly that such a type must implement the mapping protocol if it wants slice subscripting. If this makes any sense at all, I will open an issue. Eli ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Assertion in _PyManagedBuffer_FromObject()
Stefan Behnel stefan...@behnel.de wrote: Yes, that's a suitable example. It would take the ndarray out of the loop - after all, it has nothing to do with what the memoryview wants, and won't need to do any cleanup for the memoryview's buffer view either. Keeping it explicitly alive in the memoryview is just a waste of resources. Yes, this should be supported. The cleanup handler in the earlier example got me on the wrong track, that's why I kept insisting this wasn't necessary. I'm not using this anywhere. My guess is that it would be more of a feature than something to provide legacy code support for, but I can't speak for anyone else. In general, the NumPy mailing list is a good place to ask about these things. NumPy re-exports, this was confirmed in issue #10181. That's actually the main reason why I considered re-exporting rather than redirecting the standard model and built the test suite around it. Stefan Krah ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] slice subscripts for sequences and mappings
This doesn't align well with the documentation, in at least two places. snip Another place is in http://docs.python.org/dev/reference/datamodel.html: object.__getitem__(self, key) Called to implement evaluation of self[key]. For sequence types, the accepted keys should be integers and slice objects. [...] Once again, at the C API level this isn't accurate since only integer keys are handled by the sequence protocol, leaving slice keys to the mapping protocol. The datamodel doc should stay as it is, because it's correct for Python-written classes. But the relevant C API sections really need some clarification. Eli ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Assertion in _PyManagedBuffer_FromObject()
Nick Coghlan, 03.03.2012 00:49: On Sat, Mar 3, 2012 at 3:14 AM, Stefan Behnel wrote: Stefan Krah, 02.03.2012 17:42: The reason why this scheme was not chosen for a chain of memoryviews was that 'exporter' (in theory) could implement a slideshow of buffers, which means that in the face of redirecting requests m might not be equal to nd. Right. Then it's only safe when the intermediate provider knows what the underlying buffer providers do. Not unlikely in an application setting, though, and it could just be an option at creation time to activate the delegation for the ndarray above. OK, my take on the discussion so far: 1. assert() is the wrong tool for this job Absolutely. 2. the current check is too strict (it should just check for obj != NULL, not obj == exporter) I don't know. The documentation isn't very clear on the cases where obj may be NULL. Definitely on error, ok, but otherwise, the bf_getbuffer() docs do not explicitly say that it must not be NULL (they just mention a standard case): http://docs.python.org/dev/c-api/typeobj.html#buffer-object-structures and the Py_buffer docs say explicitly that the field either refers to the exporter or is NULL, without saying if this has any implications or specific meaning: http://docs.python.org/dev/c-api/buffer.html#Py_buffer Personally, I don't see a NULL (or None) value being a problem - it would just mean that the buffer does not need any release call (i.e. no cleanup), e.g. because it was statically allocated in an extension module. PyBuffer_Release() has the appropriate checks in place anyway. But I don't care either way, as long as it's documented. Stefan ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] slice subscripts for sequences and mappings
Eli Bendersky, 03.03.2012 09:36: I find a strange discrepancy in Python with regards to slice subscripting of objects, at the C API level. I mean things like obj[start:end:step]. I'd expect slice subscripts to be part of the sequence interface, and yet they are not. In fact, they are part of the mapping interface. For example, the list object has its slice get/set methods assigned to a PyMappingMethods struct. So does a bytes object, and pretty much every other object that wants to support subscripts. This doesn't align well with the documentation, in at least two places. 1) The library documentation (http://docs.python.org/dev/library/stdtypes.html) in 4.8 says: Mappings are mutable objects. There is currently only one standard mapping type, the dictionary Why then does a list implement the mapping interface? Moreover, why does bytes, an immutable object, implement the mapping interface? I think that's (partly?) for historical reasons. Originally, there were the slicing functions as part of the sequence interface. They took a start and an end index of the slice. Then, extended slicing was added to the language, and that used a slice object, which didn't fit into the sequence slicing interface. So the interface was unified using the existing mapping getitem interface, and the sequence slicing functions were eventually deprecated and removed in Py3. Stefan ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 414
Lennart Regebro regebro at gmail.com writes: So the question is if you have any proposal that is *less* confusing while still being practical. Because we do need to distinguish between binary, Unicode and native strings. Isn't this the least confusing solution? It's a matter of the degree of confusion caused (hard to assess) and also a question of taste, so there will be differing views on this. Considering use of unicode_literals, 'xxx' for text, b'yyy' for bytes and with a function wrapper to mark native strings, it becomes clear that the native strings are special cases - much less encountered when looking at code compared to 'xxx' / b'yyy', so there are fewer opportunities for confusion. Where native strings need to be discussed, then it is not unexceptional, nor I believe incorrect, to explain that they are there to suit the requirements of legacy APIs which pre-date Python 3 and the latest versions of Python 2. In terms of practicality, it is IMO quite practical (assuming 2.5 / earlier support can be dropped) to move to a 2.6+/3.x-friendly codebase, e.g. by using Armin's python-modernize. Regards, Vinay Sajip ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 414
On Sat, Mar 3, 2012 at 10:26, Vinay Sajip vinay_sa...@yahoo.co.uk wrote: Lennart Regebro regebro at gmail.com writes: So the question is if you have any proposal that is *less* confusing while still being practical. Because we do need to distinguish between binary, Unicode and native strings. Isn't this the least confusing solution? It's a matter of the degree of confusion caused (hard to assess) and also a question of taste, so there will be differing views on this. Considering use of unicode_literals, 'xxx' for text, b'yyy' for bytes and with a function wrapper to mark native strings, it becomes clear that the native strings are special cases - much less encountered when looking at code compared to 'xxx' / b'yyy', I'm not sure that's true at all. In most cases where you support both Python 2 and Python 3, most strings will be native, ie, without prefix in either Python 2 or Python 3. The native case is the most common case. In terms of practicality, it is IMO quite practical (assuming 2.5 / earlier support can be dropped) to move to a 2.6+/3.x-friendly codebase, e.g. by using Armin's python-modernize. I think there is some misunderstanding here. The binary/unicode/native separation is only possible on Python 2.6 and 2.7 at the moment, unless you use function wrappers like b(). //Lennart ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 414
Lennart Regebro regebro at gmail.com writes:
I'm not sure that's true at all. In most cases where you support both
Python 2 and Python 3, most strings will be native, ie, without
prefix in either Python 2 or Python 3. The native case is the most
common case.
Sorry, I didn't make myself clear. If you import unicode_literals, then in both
2.x and 3.x code, an unadorned literal string is text, and a b-adorned literal
string is bytes. My assertion was based on that assumption - the text (Unicode)
case then becomes the most common case.
In terms of practicality, it is
IMO quite practical (assuming 2.5 / earlier support can be dropped) to
move to a
2.6+/3.x-friendly codebase, e.g. by using Armin's python-modernize.
I think there is some misunderstanding here. The binary/unicode/native
separation is only possible on Python 2.6 and 2.7 at the moment,
unless you use function wrappers like b().
Right, and that is a possible option for 2.5 and earlier: though obviously not a
desirable one from an aesthetic point of view!
What I meant (and should have said) was: if you can drop support for 2.5 /
earlier, a lib2to3 fixer-based approach brings your 2.x code into the 3-friendly
region of 2.x - 2.6 and 2.7. You can then, using the unicode_literals approach,
arrive at a common codebase for 2.6+ and 3.x which is not slow to run (see my
other post on ported Django test run performance), and clean (looks just like 3
code, pretty much, and means the same, as far as string literals are concerned).
Where you hit native string requirements, apply the wrapper.
I don't actually use python-modernize, as I independently developed fixers when
doing the Django port late last year. I initially wrote a fixer to transform
u'xxx' to u('xxx') (as I was assuming 2.5 support was needed), and then, when it
appeared likely that Django would drop 2.5 support after 1.4, I wrote a fixer to
go from u('xxx') to 'xxx'. Once I learned to use lib2to3, with a few pointers
from Benjamin, it worked like a charm for me.
Regards,
Vinay Sajip
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe:
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Assertion in _PyManagedBuffer_FromObject()
Stefan Behnel stefan...@behnel.de wrote: 1. assert() is the wrong tool for this job Absolutely. I disagree. This assert() is meant for extension authors and not end users. I can't see how a reasonable release procedure would fail to trigger the assert(). My procedure as a C extension author is to test against a new Python version and *then* set the PyPI classifier for that version. If I download a C extension that doesn't have the 3.3 classifier set, then as a user I would not be upset if the extension throws an assert or, as Thomas Wouters pointed out, continues to work as before if not compiled in debug mode. 2. the current check is too strict (it should just check for obj != NULL, not obj == exporter) I don't know. The documentation isn't very clear on the cases where obj may be NULL. Definitely on error, ok, but otherwise, the bf_getbuffer() docs do not explicitly say that it must not be NULL (they just mention a standard case): http://docs.python.org/dev/c-api/typeobj.html#buffer-object-structures How about this: The value of view.obj is the equivalent of the return value of any C-API function that returns a new reference. The value must be NULL on error or a valid new reference to an exporting object. For a chain or a tree of views, there are two possible schemes: 1) Re-export: Each member of the tree pretends to be the exporting object and sets view.obj to a new reference to itself. 2) Redirect: The buffer request is redirected to the root object of the tree. Here view.obj will be a reference to the root object. I think it's better not to complicate this familiar scheme of owning a reference by allowing view.obj==NULL for the general case. view.obj==NULL was introduced for temporary wrapping of ad-hoc memoryviews via PyBuffer_FillInfo() and now also PyMemoryView_FromMemory(). That's why I explicitly wrote the following in the documentation of PyBuffer_FillInfo(): If this function is used as part of a getbufferproc, exporter MUST be set to the exporting object. Otherwise, exporter MUST be NULL. Stefan Krah ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] slice subscripts for sequences and mappings
On Sat, Mar 3, 2012 at 11:24, Stefan Behnel stefan...@behnel.de wrote: Eli Bendersky, 03.03.2012 09:36: I find a strange discrepancy in Python with regards to slice subscripting of objects, at the C API level. I mean things like obj[start:end:step]. I'd expect slice subscripts to be part of the sequence interface, and yet they are not. In fact, they are part of the mapping interface. For example, the list object has its slice get/set methods assigned to a PyMappingMethods struct. So does a bytes object, and pretty much every other object that wants to support subscripts. This doesn't align well with the documentation, in at least two places. 1) The library documentation (http://docs.python.org/dev/library/stdtypes.html) in 4.8 says: Mappings are mutable objects. There is currently only one standard mapping type, the dictionary Why then does a list implement the mapping interface? Moreover, why does bytes, an immutable object, implement the mapping interface? I think that's (partly?) for historical reasons. Originally, there were the slicing functions as part of the sequence interface. They took a start and an end index of the slice. Then, extended slicing was added to the language, and that used a slice object, which didn't fit into the sequence slicing interface. So the interface was unified using the existing mapping getitem interface, and the sequence slicing functions were eventually deprecated and removed in Py3. This make sense. Not that now there's also duplication in almost all objects because the mapping protocol essentially supersedes the sequence protocol for accessing elements. I.e. sq_item and sq_ass_item are no longer needed if an object implements the mapping protocol, because the mapping interface has precedence, and mp_subscript mp_ass_subscript are called instead, respectively. Because of that, the first thing they do is check whether the index is a simple number and do the work of their sequence protocol cousins. This duplicates code in almost all objects that need to support __getitem__. Eli ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] slice subscripts for sequences and mappings
Hi, I'd expect slice subscripts to be part of the sequence interface, and yet they are not. In fact, they are part of the mapping interface. For example, the list object has its slice get/set methods assigned to a PyMappingMethods struct. So does a bytes object, and pretty much every other object that wants to support subscripts. It comes from: http://hg.python.org/cpython/rev/245224d1b8c9 http://bugs.python.org/issue400998 Written by Michael Hudson and reviewed by Guido. I wonder why this patch chose to add mapping protocol support to tuples and lists, rather than add a tp_ slot for extended slicing. Regards Antoine. ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 414
On Sat, Mar 3, 2012 at 11:39, Vinay Sajip vinay_sa...@yahoo.co.uk wrote: Sorry, I didn't make myself clear. If you import unicode_literals, then in both 2.x and 3.x code, an unadorned literal string is text, and a b-adorned literal string is bytes. My assertion was based on that assumption - the text (Unicode) case then becomes the most common case. Absolutely. I think there is some misunderstanding here. The binary/unicode/native separation is only possible on Python 2.6 and 2.7 at the moment, unless you use function wrappers like b(). Right, and that is a possible option for 2.5 and earlier: though obviously not a desirable one from an aesthetic point of view! What I meant (and should have said) was: if you can drop support for 2.5 / earlier, a lib2to3 fixer-based approach brings your 2.x code into the 3-friendly region of 2.x - 2.6 and 2.7. You can then, using the unicode_literals approach, arrive at a common codebase for 2.6+ and 3.x which is not slow to run (see my other post on ported Django test run performance), and clean (looks just like 3 code, pretty much, and means the same, as far as string literals are concerned). Where you hit native string requirements, apply the wrapper. Yes, that's a doable solution. Just as the common solution of using b() and u() wrappers. But these are still more confusing and less aesthetically pleasing (and insignificantly slower) than supporting u'' in Python 3. //Lennart ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] slice subscripts for sequences and mappings
I'd expect slice subscripts to be part of the sequence interface, and yet they are not. In fact, they are part of the mapping interface. For example, the list object has its slice get/set methods assigned to a PyMappingMethods struct. So does a bytes object, and pretty much every other object that wants to support subscripts. It comes from: http://hg.python.org/cpython/rev/245224d1b8c9 http://bugs.python.org/issue400998 Written by Michael Hudson and reviewed by Guido. I wonder why this patch chose to add mapping protocol support to tuples and lists, rather than add a tp_ slot for extended slicing. Why a separate tp_ slot for extended slicing? ISTM slicing pertains to sequences, similarly to other numeric indices. If you look at PySequenceMethods it has these (apparently no longer used fields): void *was_sq_slice; void *was_sq_ass_slice; These were simple slices (pairs of numbers). I suppose if any change is considered, these fields can be re-incarnated to accept PyObject* slices similarly to the current mp_subscript and mp_ass_subscript. Eli ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Assertion in _PyManagedBuffer_FromObject()
Nick Coghlan ncogh...@gmail.com wrote:
2. the current check is too strict (it should just check for obj !=
NULL, not obj == exporter)
Yes. For anyone who is interested, see issue #14181.
3. the current check is in the wrong place (it should be in
PyObject_GetBuffer)
Agreed, since it's not memoryview specific. But I don't think we even
need to check for obj != NULL. view.obj was undocumented, and since 3.0
Include/object.h contains this:
typedef struct bufferinfo {
void *buf;
PyObject *obj;/* owned reference */
So it would be somewhat audacious to set this field to NULL. But even if
existing code uses the view.obj==NULL scheme from PyBuffer_FillInfo()
correctly, it will still work in the new implementation.
I'd just prefer to forbid this in the documentation, because it's much
easier to remember: getbuffer returns a new reference or NULL.
Stefan Krah
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe:
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] slice subscripts for sequences and mappings
Le samedi 03 mars 2012 à 14:41 +0200, Eli Bendersky a écrit : I'd expect slice subscripts to be part of the sequence interface, and yet they are not. In fact, they are part of the mapping interface. For example, the list object has its slice get/set methods assigned to a PyMappingMethods struct. So does a bytes object, and pretty much every other object that wants to support subscripts. It comes from: http://hg.python.org/cpython/rev/245224d1b8c9 http://bugs.python.org/issue400998 Written by Michael Hudson and reviewed by Guido. I wonder why this patch chose to add mapping protocol support to tuples and lists, rather than add a tp_ slot for extended slicing. Why a separate tp_ slot for extended slicing? ISTM slicing pertains to sequences, similarly to other numeric indices. If you look at PySequenceMethods it has these (apparently no longer used fields): Yes, I meant sq_ slot, my bad. Regards Antoine. ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
[Python-Dev] Why does Mac OS X python share site-packages with apple python?
On my Mac OS X 10.7.3 System I have lots of python kits installed for developing extensions. I'll just noticed that Python.org 2.7.2 uses the sames site-packages folder with Apple's 2.7.1. Since extensions compiled against Apple's 2.7.1 segv when used by python.org's 2.7.2 this is at least unfortunate. Here is the what is in sys.path for both versions. Notice /Library/Python/2.7/site-packages is in both. $ /usr/bin/python -c 'import sys,pprint; pprint.pprint( sys.path )' ['', '/usr/local/lib/wxPython-unicode-2.8.12.1/lib/python2.7/site-packages', '/usr/local/lib/wxPython-unicode-2.8.12.1/lib/python2.7/site-packages/wx-2.8-mac-unicode', '/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python27.zip', '/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7', '/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/plat-darwin', '/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/plat-mac', '/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/plat-mac/lib-scriptpackages', '/System/Library/Frameworks/Python.framework/Versions/2.7/Extras/lib/python', '/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/lib-tk', '/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/lib-old', '/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/lib-dynload', '/System/Library/Frameworks/Python.framework/Versions/2.7/Extras/lib/python/PyObjC', '/Library/Python/2.7/site-packages', '/usr/local/lib/wxPython-unicode-2.8.12.1/lib/python2.7'] $ /usr/local/bin/python2.7 -c 'import sys,pprint; pprint.pprint( sys.path )' ['', '/usr/local/lib/wxPython-unicode-2.8.12.1/lib/python2.7/site-packages', '/usr/local/lib/wxPython-unicode-2.8.12.1/lib/python2.7/site-packages/wx-2.8-mac-unicode', '/usr/local/lib/wxPython-unicode-2.8.12.1/lib/python2.7', '/Library/Python/2.7/site-packages', '/Library/Frameworks/Python.framework/Versions/2.7/lib/python27.zip', '/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7', '/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/plat-darwin', '/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/plat-mac', '/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/plat-mac/lib-scriptpackages', '/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/lib-tk', '/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/lib-old', '/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/lib-dynload', '/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages'] Barry ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] cpython compilation error
Ejaj Hassan ejjyrex at gmail.com writes: I was compiling Pcbuild.sln from cpython in vc++ 2008 and i got the error as Solution folders are not supported in this version of application-Solution folder will be displayed as unavailable. Could someone please tell me the source and reason for this error. It's because you're using the free Express edition of Visual Studio, and not the full, paid-for Visual Studio. However, I believe you can ignore the error, and the solution should still be built. (I'm not absolutely sure, as I use the full Visual Studio). Regards, Vinay Sajip ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] slice subscripts for sequences and mappings
On Sat, Mar 3, 2012 at 4:20 AM, Antoine Pitrou solip...@pitrou.net wrote: I'd expect slice subscripts to be part of the sequence interface, and yet they are not. In fact, they are part of the mapping interface. For example, the list object has its slice get/set methods assigned to a PyMappingMethods struct. So does a bytes object, and pretty much every other object that wants to support subscripts. It comes from: http://hg.python.org/cpython/rev/245224d1b8c9 http://bugs.python.org/issue400998 Written by Michael Hudson and reviewed by Guido. I wonder why this patch chose to add mapping protocol support to tuples and lists, rather than add a tp_ slot for extended slicing. That's long ago... IIRC it was for binary compatibility -- I didn't want to add an extra slot to the sq struct because it would require recompilation of 3rd party extensions. At the time that was an important concern. -- --Guido van Rossum (python.org/~guido) ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] slice subscripts for sequences and mappings
On Sat, Mar 3, 2012 at 19:58, Guido van Rossum gu...@python.org wrote:
On Sat, Mar 3, 2012 at 4:20 AM, Antoine Pitrou solip...@pitrou.net wrote:
I'd expect slice subscripts to be part of the sequence interface, and
yet they are not. In fact, they are part of the mapping interface. For
example, the list object has its slice get/set methods assigned to a
PyMappingMethods struct. So does a bytes object, and pretty much every
other object that wants to support subscripts.
It comes from:
http://hg.python.org/cpython/rev/245224d1b8c9
http://bugs.python.org/issue400998
Written by Michael Hudson and reviewed by Guido.
I wonder why this patch chose to add mapping protocol support to tuples
and lists, rather than add a tp_ slot for extended slicing.
That's long ago... IIRC it was for binary compatibility -- I didn't
want to add an extra slot to the sq struct because it would require
recompilation of 3rd party extensions. At the time that was an
important concern.
Perhaps the situation can be fixed now without binary compatibility
concerns. PySequenceMethods is:
typedef struct {
lenfunc sq_length;
binaryfunc sq_concat;
ssizeargfunc sq_repeat;
ssizeargfunc sq_item;
void *was_sq_slice;
ssizeobjargproc sq_ass_item;
void *was_sq_ass_slice;
objobjproc sq_contains;
binaryfunc sq_inplace_concat;
ssizeargfunc sq_inplace_repeat;
} PySequenceMethods;
The slots was_sq_slice and was_sq_ass_slice aren't used any
longer. These can be re-incarnated to accept a slice object, and
sequence objects can be rewritten to use them instead of implementing
the mapping protocol (is there any reason listobject implements the
mapping protocol, other than to gain the ability to use slices for
__getitem__?). Existing 3rd party extensions don't *need* to be
recompiled or changed, however. They *can* be, if their authors are
interested, of course.
Eli
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe:
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Sandboxing Python
Hi Victor, On Thu, Mar 1, 2012 at 22:59, Victor Stinner victor.stin...@gmail.com wrote: I challenge anymore to break pysandbox! I would be happy if anyone breaks it because it would make it more stronger. I tried to run the files from Lib/test/crashers and --- kind of obviously --- I found at least two of them that still segfaults execfile.py, sometimes with minor edits and sometimes directly, on CPython 2.7. As usual, I don't see the point of challenging us when we have crashers already documented. Also, it's not like Lib/test/crashers contains in detail *all* crashers that exist; some of them are of the kind there is a general issue with xxx, here is an example. If you are not concerned about segfaults but only real attacks, then fine, I will not spend the hours necessary to turn the segfault into a real attack :-) A bientôt, Armin. ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Assertion in _PyManagedBuffer_FromObject()
On Sat, Mar 3, 2012 at 03:08, Stefan Krah ste...@bytereef.org wrote: Stefan Behnel stefan...@behnel.de wrote: 1. assert() is the wrong tool for this job Absolutely. I disagree. This assert() is meant for extension authors and not end users. I can't see how a reasonable release procedure would fail to trigger the assert(). My procedure as a C extension author is to test against a new Python version and *then* set the PyPI classifier for that version. Do you test against pydebug builds of Python, or otherwise a build that actually enables asserts? Because I suspect most people don't, so they don't trigger the assert. Python is normally (that is, a release build on Windows or a regular, non-pydebug build on the rest) built without asserts. Asserts are disabled by the NDEBUG symbol, which Python passes for regular builds. Even that aside, asserts are for internal invariants, not external ones. You can use asserts in your extension module to check that your own code is passing what you think it should pass, but you shouldn't really use them to check that a library or API you use is, and Python certainly shouldn't be using it to check what code outside of the core is giving it. Aborting (which is what failed asserts do) is just not the right thing to do. If I download a C extension that doesn't have the 3.3 classifier set, then as a user I would not be upset if the extension throws an assert or, as Thomas Wouters pointed out, continues to work as before if not compiled in debug mode. 2. the current check is too strict (it should just check for obj != NULL, not obj == exporter) I don't know. The documentation isn't very clear on the cases where obj may be NULL. Definitely on error, ok, but otherwise, the bf_getbuffer() docs do not explicitly say that it must not be NULL (they just mention a standard case): http://docs.python.org/dev/c-api/typeobj.html#buffer-object-structures How about this: The value of view.obj is the equivalent of the return value of any C-API function that returns a new reference. The value must be NULL on error or a valid new reference to an exporting object. For a chain or a tree of views, there are two possible schemes: 1) Re-export: Each member of the tree pretends to be the exporting object and sets view.obj to a new reference to itself. 2) Redirect: The buffer request is redirected to the root object of the tree. Here view.obj will be a reference to the root object. I think it's better not to complicate this familiar scheme of owning a reference by allowing view.obj==NULL for the general case. view.obj==NULL was introduced for temporary wrapping of ad-hoc memoryviews via PyBuffer_FillInfo() and now also PyMemoryView_FromMemory(). That's why I explicitly wrote the following in the documentation of PyBuffer_FillInfo(): If this function is used as part of a getbufferproc, exporter MUST be set to the exporting object. Otherwise, exporter MUST be NULL. Stefan Krah ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/thomas%40python.org -- Thomas Wouters tho...@python.org Hi! I'm a .signature virus! copy me into your .signature file to help me spread! ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] slice subscripts for sequences and mappings
On Sat, Mar 3, 2012 at 10:18, Eli Bendersky eli...@gmail.com wrote:
On Sat, Mar 3, 2012 at 19:58, Guido van Rossum gu...@python.org wrote:
On Sat, Mar 3, 2012 at 4:20 AM, Antoine Pitrou solip...@pitrou.net
wrote:
I'd expect slice subscripts to be part of the sequence interface, and
yet they are not. In fact, they are part of the mapping interface. For
example, the list object has its slice get/set methods assigned to a
PyMappingMethods struct. So does a bytes object, and pretty much every
other object that wants to support subscripts.
It comes from:
http://hg.python.org/cpython/rev/245224d1b8c9
http://bugs.python.org/issue400998
Written by Michael Hudson and reviewed by Guido.
I wonder why this patch chose to add mapping protocol support to tuples
and lists, rather than add a tp_ slot for extended slicing.
That's long ago... IIRC it was for binary compatibility -- I didn't
want to add an extra slot to the sq struct because it would require
recompilation of 3rd party extensions. At the time that was an
important concern.
Perhaps the situation can be fixed now without binary compatibility
concerns. PySequenceMethods is:
typedef struct {
lenfunc sq_length;
binaryfunc sq_concat;
ssizeargfunc sq_repeat;
ssizeargfunc sq_item;
void *was_sq_slice;
ssizeobjargproc sq_ass_item;
void *was_sq_ass_slice;
objobjproc sq_contains;
binaryfunc sq_inplace_concat;
ssizeargfunc sq_inplace_repeat;
} PySequenceMethods;
The slots was_sq_slice and was_sq_ass_slice aren't used any
longer. These can be re-incarnated to accept a slice object, and
sequence objects can be rewritten to use them instead of implementing
the mapping protocol (is there any reason listobject implements the
mapping protocol, other than to gain the ability to use slices for
__getitem__?). Existing 3rd party extensions don't *need* to be
recompiled or changed, however. They *can* be, if their authors are
interested, of course.
Why even have separate tp_as_sequence and tp_as_mapping anymore? That
particular distinction never existed for Python types, so why should it
exist for C types at all? I forget if there was ever a real point to it,
but all it seems to do now is create confusion, what with many sequence
types implementing both, and PyMapping_Check() and PySequence_Check() doing
seemingly random things to come up with somewhat sensible answers. Do note
that the dict type actually implements tp_as_sequence (in order to support
containtment tests) and that PySequence_Check() has to explicitly return 0
for dicts -- which means that it will give the wrong answer for another
type that behaves exactly like dicts.
Getting rid of the misleading distinction seems like a much better idea
than trying to re-conflate some of the issues.
--
Thomas Wouters tho...@python.org
Hi! I'm a .signature virus! copy me into your .signature file to help me
spread!
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe:
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] slice subscripts for sequences and mappings
On Sat, 3 Mar 2012 12:59:13 -0800
Thomas Wouters tho...@python.org wrote:
Why even have separate tp_as_sequence and tp_as_mapping anymore? That
particular distinction never existed for Python types, so why should it
exist for C types at all? I forget if there was ever a real point to it,
but all it seems to do now is create confusion, what with many sequence
types implementing both, and PyMapping_Check() and PySequence_Check() doing
seemingly random things to come up with somewhat sensible answers.
Ironically, most of the confusion stems from sequence types
implementing the mapping protocol for extended slicing.
Do note
that the dict type actually implements tp_as_sequence (in order to support
containtment tests) and that PySequence_Check() has to explicitly return 0
for dicts -- which means that it will give the wrong answer for another
type that behaves exactly like dicts.
It seems to be a leftover:
int
PySequence_Check(PyObject *s)
{
if (PyDict_Check(s))
return 0;
return s != NULL s-ob_type-tp_as_sequence
s-ob_type-tp_as_sequence-sq_item != NULL;
}
Dict objects have a NULL sq_item so even removing the explicit check
would still return the right answer.
Getting rid of the misleading distinction seems like a much better idea
than trying to re-conflate some of the issues.
This proposal sounds rather backwards, given that we now have separate
Mapping and Sequence ABCs.
Regards
Antoine.
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe:
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] slice subscripts for sequences and mappings
Thomas Wouters, 03.03.2012 21:59: Why even have separate tp_as_sequence and tp_as_mapping anymore? That particular distinction never existed for Python types, so why should it exist for C types at all? I forget if there was ever a real point to it, but all it seems to do now is create confusion, what with many sequence types implementing both, and PyMapping_Check() and PySequence_Check() doing seemingly random things to come up with somewhat sensible answers. Do note that the dict type actually implements tp_as_sequence (in order to support containtment tests) and that PySequence_Check() has to explicitly return 0 for dicts -- which means that it will give the wrong answer for another type that behaves exactly like dicts. Getting rid of the misleading distinction seems like a much better idea than trying to re-conflate some of the issues. We're too far away from the release of Python 4 to change something with that kind of impact, though. Stefan ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Sandboxing Python
Hi, Le 03/03/2012 20:13, Armin Rigo a écrit : I challenge anymore to break pysandbox! I would be happy if anyone breaks it because it would make it more stronger. I tried to run the files from Lib/test/crashers and --- kind of obviously --- I found at least two of them that still segfaults execfile.py, sometimes with minor edits and sometimes directly, on CPython 2.7. As described in the README file of pysandbox, pysandbox doesn't protect against vulnerabilities or bugs in Python. As usual, I don't see the point of challenging us when we have crashers already documented. Also, it's not like Lib/test/crashers contains in detail *all* crashers that exist; some of them are of the kind there is a general issue with xxx, here is an example. If you are not concerned about segfaults but only real attacks, then fine, I will not spend the hours necessary to turn the segfault into a real attack :-) You may be able to exploit crashers, but I don't plan to workaround such CPython bug in pysandbox. I'm looking for vulnerabilities in pysandbox, not in CPython. Victor ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Why does Mac OS X python share site-packages with apple python?
In article 5a0e2490-a743-4729-a752-d94524ea9...@barrys-emacs.org, Barry Scott ba...@barrys-emacs.org wrote: On my Mac OS X 10.7.3 System I have lots of python kits installed for developing extensions. I'll just noticed that Python.org 2.7.2 uses the sames site-packages folder with Apple's 2.7.1. Since extensions compiled against Apple's 2.7.1 segv when used by python.org's 2.7.2 this is at least unfortunate. Here is the what is in sys.path for both versions. Notice /Library/Python/2.7/site-packages is in both. That directory is in the default sys.path for both the Apple-supplied Python 2.7 in Lion and for the python.org Python 2.7's but that doesn't mean both versions use the same site-packages directory: $ /usr/bin/python2.7 -c import distutils.sysconfig; \ print(distutils.sysconfig.get_python_lib()) /Library/Python/2.7/site-packages $ /usr/local/bin/python2.7 -c import distutils.sysconfig; \ print(distutils.sysconfig.get_python_lib()) /Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-pack ages That means that, by default, packages installed by Distutils-based installs (setup.py, easy_install, pip, et al) will be installed to the corresponding directory for each version. The python.org OS X Pythons (and built-from-source framework builds) add the Apple-specific directory to the search path in order to allow sharing of installed third-party packages between the two. The feature was added in 2.7 and 3.1+ and tracked in Issue4865 (http://bugs.python.org/issue4865). Please open a new issue on the tracker if you have examples of how this is causing problems. Thanks. -- Ned Deily, n...@acm.org ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 414
2to3 should recognize the str(string_literal) (or nstr(), or native(), etc) as a native string and does not add prefix u to it. And you have to explicitly specify these tips. That is already implemented. 2to3 *never* adds a u prefix anywhere, including not for str(string_literal). Regards, Martin ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 416: Add a frozendict builtin type
Le 29/02/2012 19:21, Victor Stinner a écrit : Rationale = (...) Use cases of frozendict: (...) I updated the PEP to list use cases described in the other related mailing list thread. --- Use cases: * frozendict lookup can be done at compile time instead of runtime because the mapping is read-only. frozendict can be used instead of a preprocessor to remove conditional code at compilation, like code specific to a debug build. * hashable frozendict can be used as a key of a mapping or as a member of set. frozendict can be used to implement a cache. * frozendict avoids the need of a lock when the frozendict is shared by multiple threads or processes, especially hashable frozendict. It would also help to prohibe coroutines (generators + greenlets) to modify the global state. * frozendict helps to implement read-only object proxies for security modules. For example, it would be possible to use frozendict type for __builtins__ mapping or type.__dict__. This is possible because frozendict is compatible with the PyDict C API. * frozendict avoids the need of a read-only proxy in some cases. frozendict is faster than a proxy because getting an item in a frozendict is a fast lookup whereas a proxy requires a function call. * use a frozendict as the default value of function argument: avoid the problem of mutable default argument. --- Victor ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] cpython compilation error
2012/3/3 Vinay Sajip vinay_sa...@yahoo.co.uk Ejaj Hassan ejjyrex at gmail.com writes: I was compiling Pcbuild.sln from cpython in vc++ 2008 and i got the error as Solution folders are not supported in this version of application-Solution folder will be displayed as unavailable. Could someone please tell me the source and reason for this error. It's because you're using the free Express edition of Visual Studio, and not the full, paid-for Visual Studio. However, I believe you can ignore the error, and the solution should still be built. (I'm not absolutely sure, as I use the full Visual Studio). I confirm: you can safely ignore this warning message. The Solution folder is a convenient place to group files not related to a sub-project, like the readme.txt file. It has no effect on the build. -- Amaury Forgeot d'Arc ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Why does Mac OS X python share site-packages with apple python?
On 3 Mar 2012, at 21:57, Ned Deily wrote: In article 5a0e2490-a743-4729-a752-d94524ea9...@barrys-emacs.org, Barry Scott ba...@barrys-emacs.org wrote: On my Mac OS X 10.7.3 System I have lots of python kits installed for developing extensions. I'll just noticed that Python.org 2.7.2 uses the sames site-packages folder with Apple's 2.7.1. Since extensions compiled against Apple's 2.7.1 segv when used by python.org's 2.7.2 this is at least unfortunate. Here is the what is in sys.path for both versions. Notice /Library/Python/2.7/site-packages is in both. That directory is in the default sys.path for both the Apple-supplied Python 2.7 in Lion and for the python.org Python 2.7's but that doesn't mean both versions use the same site-packages directory: $ /usr/bin/python2.7 -c import distutils.sysconfig; \ print(distutils.sysconfig.get_python_lib()) /Library/Python/2.7/site-packages $ /usr/local/bin/python2.7 -c import distutils.sysconfig; \ print(distutils.sysconfig.get_python_lib()) /Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-pack ages That means that, by default, packages installed by Distutils-based installs (setup.py, easy_install, pip, et al) will be installed to the corresponding directory for each version. The python.org OS X Pythons (and built-from-source framework builds) add the Apple-specific directory to the search path in order to allow sharing of installed third-party packages between the two. The feature was added in 2.7 and 3.1+ and tracked in Issue4865 (http://bugs.python.org/issue4865). Please open a new issue on the tracker if you have examples of how this is causing problems. Thanks. Yes I have a example that SEGV, pysvn details of kit location in bug report. I take it that any .so will crash as well. Only .py can be shared. http://bugs.python.org/issue14188 Look at the order of the sys.path the apple python site-packages hides the python.org site-packages. If the shared folder was after the python.org then imports could be made to work. Barry ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Sandboxing Python
On Sat, Mar 3, 2012 at 1:37 PM, Victor Stinner victor.stin...@gmail.com wrote: Hi, Le 03/03/2012 20:13, Armin Rigo a écrit : I challenge anymore to break pysandbox! I would be happy if anyone breaks it because it would make it more stronger. I tried to run the files from Lib/test/crashers and --- kind of obviously --- I found at least two of them that still segfaults execfile.py, sometimes with minor edits and sometimes directly, on CPython 2.7. As described in the README file of pysandbox, pysandbox doesn't protect against vulnerabilities or bugs in Python. As usual, I don't see the point of challenging us when we have crashers already documented. Also, it's not like Lib/test/crashers contains in detail *all* crashers that exist; some of them are of the kind there is a general issue with xxx, here is an example. If you are not concerned about segfaults but only real attacks, then fine, I will not spend the hours necessary to turn the segfault into a real attack :-) You may be able to exploit crashers, but I don't plan to workaround such CPython bug in pysandbox. I'm looking for vulnerabilities in pysandbox, not in CPython. Victor Well ok. But then what's the point of challenging people? You say this is secure according to my knowledge and when armin says no it's not, you claim this is the wrong kind of security exploit. Segfaults (most of them) can generally be made into arbitrary code execution, hence the pysandbox is not quite secure. Even further, any sort of this security restrictions where you modify locals globals etc. would be seriously prone to attacks like those segfaults, unless you do something with the VM you're running. This makes it slightly less convincing to argue that the VM requires new features (in this case frozendict) in order to support the kind of program that's broken in the first place. Well, I think I'm seriously missing something. Cheers, fijal ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 416: Add a frozendict builtin type
On Sat, Mar 3, 2012 at 4:11 PM, Victor Stinner victor.stin...@gmail.com wrote: Le 29/02/2012 19:21, Victor Stinner a écrit : Rationale = (...) Use cases of frozendict: (...) I updated the PEP to list use cases described in the other related mailing list thread. --- Use cases: * frozendict lookup can be done at compile time instead of runtime because the mapping is read-only. frozendict can be used instead of a preprocessor to remove conditional code at compilation, like code specific to a debug build. * hashable frozendict can be used as a key of a mapping or as a member of set. frozendict can be used to implement a cache. * frozendict avoids the need of a lock when the frozendict is shared by multiple threads or processes, especially hashable frozendict. It would also help to prohibe coroutines (generators + greenlets) to modify the global state. * frozendict helps to implement read-only object proxies for security modules. For example, it would be possible to use frozendict type for __builtins__ mapping or type.__dict__. This is possible because frozendict is compatible with the PyDict C API. * frozendict avoids the need of a read-only proxy in some cases. frozendict is faster than a proxy because getting an item in a frozendict is a fast lookup whereas a proxy requires a function call. * use a frozendict as the default value of function argument: avoid the problem of mutable default argument. Is your implementation (adapted to a standalone type) something you could put up on the cheeseshop? -eric ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] slice subscripts for sequences and mappings
On Sat, Mar 3, 2012 at 13:02, Antoine Pitrou solip...@pitrou.net wrote:
On Sat, 3 Mar 2012 12:59:13 -0800
Thomas Wouters tho...@python.org wrote:
Why even have separate tp_as_sequence and tp_as_mapping anymore? That
particular distinction never existed for Python types, so why should it
exist for C types at all? I forget if there was ever a real point to it,
but all it seems to do now is create confusion, what with many sequence
types implementing both, and PyMapping_Check() and PySequence_Check()
doing
seemingly random things to come up with somewhat sensible answers.
Ironically, most of the confusion stems from sequence types
implementing the mapping protocol for extended slicing.
Do note
that the dict type actually implements tp_as_sequence (in order to
support
containtment tests) and that PySequence_Check() has to explicitly return
0
for dicts -- which means that it will give the wrong answer for another
type that behaves exactly like dicts.
It seems to be a leftover:
int
PySequence_Check(PyObject *s)
{
if (PyDict_Check(s))
return 0;
return s != NULL s-ob_type-tp_as_sequence
s-ob_type-tp_as_sequence-sq_item != NULL;
}
Dict objects have a NULL sq_item so even removing the explicit check
would still return the right answer.
Getting rid of the misleading distinction seems like a much better idea
than trying to re-conflate some of the issues.
This proposal sounds rather backwards, given that we now have separate
Mapping and Sequence ABCs.
I'm not sure how the ABCs, which are abstract declarations of semantics,
tie into this specific implementation detail. ABCs work just as well for
Python types as for C types, and Python types don't have this distinction.
The distinction in C types has been *practically* useless for years, so why
should it stay? What is the actual benefit here?
--
Thomas Wouters tho...@python.org
Hi! I'm a .signature virus! copy me into your .signature file to help me
spread!
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe:
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] slice subscripts for sequences and mappings
On Sat, Mar 3, 2012 at 13:12, Stefan Behnel stefan...@behnel.de wrote: Thomas Wouters, 03.03.2012 21:59: Why even have separate tp_as_sequence and tp_as_mapping anymore? That particular distinction never existed for Python types, so why should it exist for C types at all? I forget if there was ever a real point to it, but all it seems to do now is create confusion, what with many sequence types implementing both, and PyMapping_Check() and PySequence_Check() doing seemingly random things to come up with somewhat sensible answers. Do note that the dict type actually implements tp_as_sequence (in order to support containtment tests) and that PySequence_Check() has to explicitly return 0 for dicts -- which means that it will give the wrong answer for another type that behaves exactly like dicts. Getting rid of the misleading distinction seems like a much better idea than trying to re-conflate some of the issues. We're too far away from the release of Python 4 to change something with that kind of impact, though. It's not hard to do this in a backward-compatible way. Either grow one of the tp_as_* to include everything a 'unified' tp_as_everything struct would need, or add a new tp_as_everything slot in the type struct. Then add a tp_flag to indicate that the type has this new layout/slot and guard all uses of the new slots with a check for that flag. If the type doesn't have the new layout or doesn't have it or the slots in it set, the code can fall back to the old try-one-and-then-the-other behaviour of dealing with tp_as_sequence and tp_as_mapping. (Let's not forget about tp_as_sequence.sq_concat, tp_as_number.nb_add, tp_as_sequence.sq_repeat and tp_as_number.nb_mul either.) -- Thomas Wouters tho...@python.org Hi! I'm a .signature virus! copy me into your .signature file to help me spread! ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Sandboxing Python
On Sat, Mar 3, 2012 at 6:02 PM, Maciej Fijalkowski fij...@gmail.com wrote: On Sat, Mar 3, 2012 at 1:37 PM, Victor Stinner victor.stin...@gmail.com wrote: Hi, Le 03/03/2012 20:13, Armin Rigo a écrit : I challenge anymore to break pysandbox! I would be happy if anyone breaks it because it would make it more stronger. I tried to run the files from Lib/test/crashers and --- kind of obviously --- I found at least two of them that still segfaults execfile.py, sometimes with minor edits and sometimes directly, on CPython 2.7. As described in the README file of pysandbox, pysandbox doesn't protect against vulnerabilities or bugs in Python. As usual, I don't see the point of challenging us when we have crashers already documented. Also, it's not like Lib/test/crashers contains in detail *all* crashers that exist; some of them are of the kind there is a general issue with xxx, here is an example. If you are not concerned about segfaults but only real attacks, then fine, I will not spend the hours necessary to turn the segfault into a real attack :-) You may be able to exploit crashers, but I don't plan to workaround such CPython bug in pysandbox. I'm looking for vulnerabilities in pysandbox, not in CPython. Victor Well ok. But then what's the point of challenging people? You say this is secure according to my knowledge and when armin says no it's not, you claim this is the wrong kind of security exploit. Segfaults (most of them) can generally be made into arbitrary code execution, hence the pysandbox is not quite secure. Even further, any sort of this security restrictions where you modify locals globals etc. would be seriously prone to attacks like those segfaults, unless you do something with the VM you're running. This makes it slightly less convincing to argue that the VM requires new features (in this case frozendict) in order to support the kind of program that's broken in the first place. Well, I think I'm seriously missing something. Could we put asserts in the places where segfaults may happen? Then Victor could say if you want this to be secure then you must build your Python executable with asserts on. IIRC some of the segfaults *already* trigger asserts when those are enabled. -- --Guido van Rossum (python.org/~guido) ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] slice subscripts for sequences and mappings
Thomas Wouters, 03.03.2012 21:59:
Why even have separate tp_as_sequence and tp_as_mapping anymore? That
particular distinction never existed for Python types, so why should it
exist for C types at all? I forget if there was ever a real point to it,
but all it seems to do now is create confusion, what with many sequence
types implementing both, and PyMapping_Check() and PySequence_Check()
doing
seemingly random things to come up with somewhat sensible answers. Do
note
that the dict type actually implements tp_as_sequence (in order to
support
containtment tests) and that PySequence_Check() has to explicitly return
0
for dicts -- which means that it will give the wrong answer for
another
type that behaves exactly like dicts.
Getting rid of the misleading distinction seems like a much better idea
than trying to re-conflate some of the issues.
We're too far away from the release of Python 4 to change something with
that kind of impact, though.
It's not hard to do this in a backward-compatible way. Either grow one of
the tp_as_* to include everything a 'unified' tp_as_everything struct would
need, or add a new tp_as_everything slot in the type struct. Then add a
tp_flag to indicate that the type has this new layout/slot and guard all
uses of the new slots with a check for that flag. If the type doesn't have
the new layout or doesn't have it or the slots in it set, the code can fall
back to the old try-one-and-then-the-other behaviour of dealing with
tp_as_sequence and tp_as_mapping.
(Let's not forget about tp_as_sequence.sq_concat, tp_as_number.nb_add,
tp_as_sequence.sq_repeat and tp_as_number.nb_mul either.)
There's nothing to unify, really, since PyMappingMethods is just a
subset of PySequenceMethods:
typedef struct {
lenfunc mp_length;
binaryfunc mp_subscript;
objobjargproc mp_ass_subscript;
} PyMappingMethods;
with the small difference that in PySequenceMethods sq_item and
sq_ass_item just accept numeric indices. However, if PySequenceMethods
has the was_sq_sclies and was_sq_ass_slice fields are reinstated to
accept a generic PyObject, PyMappingMethods will be a true subset.
If we look at the code, this becomes even clearer: in a full grep on
the Python 3.3 source, there is no object that defines tp_as_mapping
but does not also define tp_as_sequence, except Modules/_sqlite/row.c
[I'm not familiar enough with the _sqlite module, but there's a chance
it would make sense for the Row to be a sequence too].
Eli
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe:
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Sandboxing Python
On Sat, Mar 3, 2012 at 6:51 PM, Guido van Rossum gu...@python.org wrote: On Sat, Mar 3, 2012 at 6:02 PM, Maciej Fijalkowski fij...@gmail.com wrote: On Sat, Mar 3, 2012 at 1:37 PM, Victor Stinner victor.stin...@gmail.com wrote: Hi, Le 03/03/2012 20:13, Armin Rigo a écrit : I challenge anymore to break pysandbox! I would be happy if anyone breaks it because it would make it more stronger. I tried to run the files from Lib/test/crashers and --- kind of obviously --- I found at least two of them that still segfaults execfile.py, sometimes with minor edits and sometimes directly, on CPython 2.7. As described in the README file of pysandbox, pysandbox doesn't protect against vulnerabilities or bugs in Python. As usual, I don't see the point of challenging us when we have crashers already documented. Also, it's not like Lib/test/crashers contains in detail *all* crashers that exist; some of them are of the kind there is a general issue with xxx, here is an example. If you are not concerned about segfaults but only real attacks, then fine, I will not spend the hours necessary to turn the segfault into a real attack :-) You may be able to exploit crashers, but I don't plan to workaround such CPython bug in pysandbox. I'm looking for vulnerabilities in pysandbox, not in CPython. Victor Well ok. But then what's the point of challenging people? You say this is secure according to my knowledge and when armin says no it's not, you claim this is the wrong kind of security exploit. Segfaults (most of them) can generally be made into arbitrary code execution, hence the pysandbox is not quite secure. Even further, any sort of this security restrictions where you modify locals globals etc. would be seriously prone to attacks like those segfaults, unless you do something with the VM you're running. This makes it slightly less convincing to argue that the VM requires new features (in this case frozendict) in order to support the kind of program that's broken in the first place. Well, I think I'm seriously missing something. Could we put asserts in the places where segfaults may happen? Then Victor could say if you want this to be secure then you must build your Python executable with asserts on. IIRC some of the segfaults *already* trigger asserts when those are enabled. It's easy for some cases. Stack exhaustion cases might be significantly harder although you might pass some compiler-specific options to defend against that. The problem is a bit that those are examples, which mean that they might either touch specific parts of code or code that looks like that. A good example of a latter is chaining of iterators. Any iterators that can be chained can be made into a stack exhaustion segfault. I suppose with a bit of effort it might be made significantly harder though. Cheers, fijal ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] slice subscripts for sequences and mappings
On Sun, Mar 4, 2012 at 12:24 PM, Thomas Wouters tho...@python.org wrote: (Let's not forget about tp_as_sequence.sq_concat, tp_as_number.nb_add, tp_as_sequence.sq_repeat and tp_as_number.nb_mul either.) Indeed, let's not forget about those, which are a compatibility problem in and of themselves: http://bugs.python.org/issue11477 At most, the tp_mapping and tp_as_sequence overlap should be an FAQ entry in the devguide that says yes, the implementation of this is weird. It's like that for historical reasons, and fixing it is a long way down the priority list for changes Cheers, Nick. -- Nick Coghlan | ncogh...@gmail.com | Brisbane, Australia ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 414
04.03.12 00:12, Martin v. Löwis написав(ла): 2to3 should recognize the str(string_literal) (or nstr(), or native(), etc) as a native string and does not add prefix u to it. And you have to explicitly specify these tips. That is already implemented. 2to3 *never* adds a u prefix anywhere, including not for str(string_literal). Sorry, I mean *3to2*. ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
[Python-Dev] PEP 414 updated
My rewritten version of PEP 414 is now up (http://www.python.org/dev/peps/pep-0414/). It describes in detail a lot more of the historical background that was taken as read when Guido accepted the PEP. Can we let the interminable discussion die now? Please? Regards, Nick. P.S. If you find an actual factual *error* in the PEP, let me know by private email. If you just disagree with Guido's acceptance of the PEP, or want to quibble about my personal choice of wording on a particular point, please just let it rest. -- Nick Coghlan | ncogh...@gmail.com | Brisbane, Australia ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
