Re: [Python-Dev] Using PEP384 Stable ABI for the lzma extension module

2011-10-06 Thread Charles-François Natali
That's not a given. Depending on the memory allocator, a copy can be avoided. That's why the str += str hack is much more efficient under Linux than Windows, AFAIK. Even Linux will have to copy a block on realloc in certain cases, no? Probably so. How often is totally unknown

Re: [Python-Dev] Using PEP384 Stable ABI for the lzma extension module

2011-10-06 Thread Amaury Forgeot d'Arc
Le 6 octobre 2011 10:09, Charles-François Natali neolo...@free.fr a écrit : But under certain circumstances (if a large block is requested), the allocator uses mmap(), no? That's right, if the block requested is bigger than mmap_threshold (256K by default with glibc, forgetting the sliding

Re: [Python-Dev] Using PEP384 Stable ABI for the lzma extension module

2011-10-05 Thread Martin v. Löwis
Not sure what you are using it for. If you need to extend the buffer in case it is too small, there is absolutely no way this could work without copies in the general case because of how computers use address space. Even _PyBytes_Resize will copy the data. That's not a given. Depending on the

Re: [Python-Dev] Using PEP384 Stable ABI for the lzma extension module

2011-10-05 Thread Antoine Pitrou
Le mercredi 05 octobre 2011 à 18:12 +0200, Martin v. Löwis a écrit : Not sure what you are using it for. If you need to extend the buffer in case it is too small, there is absolutely no way this could work without copies in the general case because of how computers use address space. Even

Re: [Python-Dev] Using PEP384 Stable ABI for the lzma extension module

2011-10-05 Thread Toshio Kuratomi
On Wed, Oct 05, 2011 at 06:14:08PM +0200, Antoine Pitrou wrote: Le mercredi 05 octobre 2011 à 18:12 +0200, Martin v. Löwis a écrit : Not sure what you are using it for. If you need to extend the buffer in case it is too small, there is absolutely no way this could work without copies in

Re: [Python-Dev] Using PEP384 Stable ABI for the lzma extension module

2011-10-05 Thread Antoine Pitrou
On Wed, 5 Oct 2011 09:38:10 -0700 Toshio Kuratomi a.bad...@gmail.com wrote: On Wed, Oct 05, 2011 at 06:14:08PM +0200, Antoine Pitrou wrote: Le mercredi 05 octobre 2011 à 18:12 +0200, Martin v. Löwis a écrit : Not sure what you are using it for. If you need to extend the buffer in case

[Python-Dev] Using PEP384 Stable ABI for the lzma extension module

2011-10-04 Thread Amaury Forgeot d'Arc
Hi, Has someone already tried to *really* use Py_LIMITED_API for some serious extension module? I wanted to give it a try for the _lzma module (see issue 6715) because liblzma does not compile with Microsoft compilers; an alternative could be to use mingw to (pre)build _lzma.pyd, which would link

Re: [Python-Dev] Using PEP384 Stable ABI for the lzma extension module

2011-10-04 Thread Nick Coghlan
(My comments are based on the assumption Amaury started with http://hg.python.org/sandbox/nvawda/file/09d984063fca/Modules/_lzmamodule.c) On Tue, Oct 4, 2011 at 12:18 PM, Amaury Forgeot d'Arc amaur...@gmail.com wrote: - Py_LIMITED_API is incompatible with --with-pydebug, and compilation stops.  

Re: [Python-Dev] Using PEP384 Stable ABI for the lzma extension module

2011-10-04 Thread Nick Coghlan
On Tue, Oct 4, 2011 at 1:05 PM, Nick Coghlan ncogh...@gmail.com wrote: It's probably not a bad idea, otherwise we may compilation without realising it. s/may/may break/ Actually testing the ABI stability would be much harder - somehow building an extension module against 3.2 with the limited

Re: [Python-Dev] Using PEP384 Stable ABI for the lzma extension module

2011-10-04 Thread Martin v. Löwis
Amaury: thanks for your experiment and your report. - I replaced PyBytes_GET_SIZE() with Py_SIZE(), which is OK, and PyBytes_AS_STRING() with PyBytes_AsString(), which may have a slight performance impact. That's the whole point of the stable ABI: AS_STRING assumes that there is an ob_sval

Re: [Python-Dev] Using PEP384 Stable ABI for the lzma extension module

2011-10-04 Thread Martin v. Löwis
- Py_LIMITED_API is incompatible with --with-pydebug, and compilation stops. I skipped the check to continue. That seems like an odd (and undesirable) restriction. It's deliberate, though. If different Python versions are going to expose the same ABI, it seems strange of debug and release

Re: [Python-Dev] Using PEP384 Stable ABI for the lzma extension module

2011-10-04 Thread Amaury Forgeot d'Arc
2011/10/4 Martin v. Löwis mar...@v.loewis.de: - _PyBytes_Resize() is missing; I moved it under a Py_LIMITED_API   section. ??? Are you proposing to add _PyBytes_Resize to the Py_LIMITED_API set of functions? It's not even an API function in the first place (it starts with an underscore), so

Re: [Python-Dev] Using PEP384 Stable ABI for the lzma extension module

2011-10-04 Thread Antoine Pitrou
On Tue, 4 Oct 2011 13:05:58 -0400 Nick Coghlan ncogh...@gmail.com wrote: - _PyBytes_Resize() is missing; I moved it under a Py_LIMITED_API  section. No, that's not valid. Bytes are officially immutable - mutating them when the reference count is only 1 is a private for a reason. The

Re: [Python-Dev] Using PEP384 Stable ABI for the lzma extension module

2011-10-04 Thread Martin v. Löwis
Am 04.10.11 21:06, schrieb Amaury Forgeot d'Arc: 2011/10/4 Martin v. Löwismar...@v.loewis.de: - _PyBytes_Resize() is missing; I moved it under a Py_LIMITED_API section. ??? Are you proposing to add _PyBytes_Resize to the Py_LIMITED_API set of functions? It's not even an API function in

Re: [Python-Dev] Using PEP384 Stable ABI for the lzma extension module

2011-10-04 Thread Nick Coghlan
On Tue, Oct 4, 2011 at 3:12 PM, Antoine Pitrou solip...@pitrou.net wrote: Uh, no, it depends what you're doing. There's no reason not to allow people to resize a bytes object which they've just allocated and is still private to their code. That's the whole reason why _PyBytes_Resize() exists,

Re: [Python-Dev] Using PEP384 Stable ABI for the lzma extension module

2011-10-04 Thread Antoine Pitrou
On Tue, 04 Oct 2011 21:33:34 +0200 Martin v. Löwis mar...@v.loewis.de wrote: Am 04.10.11 21:06, schrieb Amaury Forgeot d'Arc: 2011/10/4 Martin v. Löwismar...@v.loewis.de: - _PyBytes_Resize() is missing; I moved it under a Py_LIMITED_API section. ??? Are you proposing to add