Re: [Python-Dev] (ctypes example) libffi embedded in CPython
On Wed, Mar 11, 2015 at 8:17 PM, Antoine Pitrou solip...@pitrou.net wrote: On Wed, 11 Mar 2015 19:05:57 +0100 Antoine Pitrou solip...@pitrou.net wrote: But they are not ctypes. For example, cffi wouldn't be obvious to use for interfacing with non-C code, since it requires you to write C-like declarations. You mean like Fortran? Or what precisely? Any toolchain that can generate native code. It can be Fortran, but it can also be code generated at runtime without there being any external declaration. Having to generate C declarations for such code would be a distraction. For instance, you can look at the compiler example that Eli wrote using llvmlite. It implements a JIT compiler for a toy language. The JIT-compiled function is then declared and called using a simple ctypes declaration: https://github.com/eliben/pykaleidoscope/blob/master/chapter7.py#L937 Regards Antoine. It might be a matter of taste, but I don't find declaring C functions any more awkward than using strange interface that ctypes comes with. the equivalent in cffi would be ffi.cast(double (*)(), x) ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] libffi embedded in CPython
On Wed, Mar 11, 2015 at 11:34 PM, Victor Stinner victor.stin...@gmail.com wrote: Le 11 mars 2015 18:29, Brett Cannon br...@python.org a écrit : I'm going to propose a somewhat controversial idea: let's deprecate the ctypes module. In the past I tried to deprecate many functions or modules because they are rarely or never used. Many developers prefered to keep them. By the way, I still want to remove plat-xxx modules like IN or CDROM :-) Getopt was deprecated when optparse was added to the stdlib. Then optparse was deprecated when argparse was added to the stdlib. Cython and cffi are not part of the stdlib and can be hard to install on some platforms. Ctypes is cool because it doesn't require C headers nor a C compiler. Is it possible to use cffi without a C compiler/headers as easily than ctypes? yes, it has two modes, one that does that and the other that does extra safety at the cost of a C compiler ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] libffi embedded in CPython
Le 11 mars 2015 18:29, Brett Cannon br...@python.org a écrit : I'm going to propose a somewhat controversial idea: let's deprecate the ctypes module. In the past I tried to deprecate many functions or modules because they are rarely or never used. Many developers prefered to keep them. By the way, I still want to remove plat-xxx modules like IN or CDROM :-) Getopt was deprecated when optparse was added to the stdlib. Then optparse was deprecated when argparse was added to the stdlib. Cython and cffi are not part of the stdlib and can be hard to install on some platforms. Ctypes is cool because it doesn't require C headers nor a C compiler. Is it possible to use cffi without a C compiler/headers as easily than ctypes? If you want to move forward, you should help to integrate cffi into the stdlib. What's the status? Victor ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] libffi embedded in CPython
On Wed, Mar 11, 2015 at 8:31 PM, Wes Turner wes.tur...@gmail.com wrote: On Mar 11, 2015 12:55 PM, Maciej Fijalkowski fij...@gmail.com wrote: On Wed, Mar 11, 2015 at 7:50 PM, Antoine Pitrou solip...@pitrou.net wrote: On Wed, 11 Mar 2015 17:27:58 + Brett Cannon br...@python.org wrote: Did anyone ever step forward to do this? I'm a bit worried about the long-term viability of ctypes if we don't have a maintainer or at least someone making sure we are staying up-to-date with upstream libffi. The ctypes module is a dangerous thing, so having a chunk of C code that isn't being properly maintained seems to me to make it even more dangerous. Depends what you call dangerous. C code doesn't rot quicker than pure Python code :-) Also, libffi really offers a wrapper around platform ABIs, which rarely change. And yet, lesser known ABIs in libffi contain bugs (as we discovered trying to work there with anything else than x86 really). Also there *are* ABI differencies that change slowly over time (e.g. requiring stack to be 16 byte aligned) Are there tests for this? What do you mean? The usual failure mode is will segfault every now and again if the moon is in the right position (e.g. the stack alignment thing only happens if the underlaying function uses certain SSE instructions that compilers emit these days in certain circumstances) ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
[Python-Dev] Why does python use relative instead of absolute path when calling LoadLibrary*
Hi, While looking at the import code of python for C extensions, I was wondering why we pass a relative path instead of an absolute path to LoadLibraryEx (see bottom for some context). In python 2.7, the full path existence was even checked before calling into LoadLibraryEx ( https://github.com/python/cpython/blob/2.7/Python/dynload_win.c#L189), but it looks like this check was removed in python 3.x branch. Is there any defined behaviour that depends on this path to be relative ? Context --- The reason why I am interested in this is the potential use of SetDllDirectory to share dlls between multiple python extensions. Currently, the only solutions I am aware of are: 1. putting the dlls in the PATH 2. bundling the dlls side by side the .pyd 3. patching packages to use preloading (using e.g. ctypes) I am investigating a solution 4, where the dlls would be put in a separate private directory only known of python itself, without the need to modify PATH. Patching python to use SetDllDirectory(some private paths specific to a python interpreter) works perfectly, except that it slightly changes the semantics of LoadLibraryEx not to look for dlls in the current directory. This breaks importing extensions built in place, unless I modify the call in ;https://github.com/python/cpython/blob/2.7/Python/dynload_win.c#L195 from: hDLL = LoadLibraryEx(pathname, NULL LOAD_WITH_ALTERED_SEARCH_PATH) to hDLL = LoadLibraryEx(pathbuf, NULL LOAD_WITH_ALTERED_SEARCH_PATH) That seems to work, but I am quite worried about changing any import semantics by accident. David ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] libffi embedded in CPython
On Wed, Mar 11, 2015 at 8:05 PM, Antoine Pitrou solip...@pitrou.net wrote: On Wed, 11 Mar 2015 19:54:58 +0200 Maciej Fijalkowski fij...@gmail.com wrote: Depends what you call dangerous. C code doesn't rot quicker than pure Python code :-) Also, libffi really offers a wrapper around platform ABIs, which rarely change. And yet, lesser known ABIs in libffi contain bugs (as we discovered trying to work there with anything else than x86 really). Also there *are* ABI differencies that change slowly over time (e.g. requiring stack to be 16 byte aligned) Well, sure. The point is, such bugs are unlikely to appear at a fast rate... Also, I don't understand why libffi issues would affect cffi any less than it affects ctypes, at least in the compiler-less mode of operation. My point here was only about shipping own libffi vs using the system one (and it does affect cffi equally with or without compiler) We now have things like cffi and Cython for people who need to interface with C code. Both of those projects are maintained. And they are not overly difficult to work with. But they are not ctypes. For example, cffi wouldn't be obvious to use for interfacing with non-C code, since it requires you to write C-like declarations. You mean like Fortran? Or what precisely? Any toolchain that can generate native code. It can be Fortran, but it can also be code generated at runtime without there being any external declaration. Having to generate C declarations for such code would be a distraction. Of course, if cffi gains the same ability as ctypes (namely to lookup a function and declare its signature without going through the FFI.cdef() interface), that issue disappears. As a side note, ctypes has a large number of users, so even if it were deprecated that wouldn't be a good reason to stop maintaining it. And calling cffi simple while it relies on a parser of the C language (which would then have to be bundled with Python) is a bit misleading IMO. Regards Antoine. ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/fijall%40gmail.com ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 488: elimination of PYO files
On Fri, Mar 6, 2015 at 6:49 PM Benjamin Peterson benja...@python.org wrote: On Fri, Mar 6, 2015, at 15:11, Brett Cannon wrote: OK, but that doesn't influence the PEP's goal of dropping .pyo files. Correct. Are you suggesting that the tag be changed to be less specific to optimizations and more free-form? Like `importlib.cpython-35.__no-asserts_no-docstrings__.pyc`? Otherwise stuff like this gets baked into the .pyc file itself instead of the file name, but I don't think we should just drop the ability to switch off asserts and docstrings like Mark seemed to be suggesting. Basically, though the filename strings could perhaps be more compact. I have a poll going on G+ to see what people think of the various proposed file name formats at https://plus.google.com/u/0/+BrettCannon/posts/fZynLNwHWGm . Feel free to vote if you have an opinion. ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] libffi embedded in CPython
On Wed, 11 Mar 2015 19:54:58 +0200 Maciej Fijalkowski fij...@gmail.com wrote: Depends what you call dangerous. C code doesn't rot quicker than pure Python code :-) Also, libffi really offers a wrapper around platform ABIs, which rarely change. And yet, lesser known ABIs in libffi contain bugs (as we discovered trying to work there with anything else than x86 really). Also there *are* ABI differencies that change slowly over time (e.g. requiring stack to be 16 byte aligned) Well, sure. The point is, such bugs are unlikely to appear at a fast rate... Also, I don't understand why libffi issues would affect cffi any less than it affects ctypes, at least in the compiler-less mode of operation. We now have things like cffi and Cython for people who need to interface with C code. Both of those projects are maintained. And they are not overly difficult to work with. But they are not ctypes. For example, cffi wouldn't be obvious to use for interfacing with non-C code, since it requires you to write C-like declarations. You mean like Fortran? Or what precisely? Any toolchain that can generate native code. It can be Fortran, but it can also be code generated at runtime without there being any external declaration. Having to generate C declarations for such code would be a distraction. Of course, if cffi gains the same ability as ctypes (namely to lookup a function and declare its signature without going through the FFI.cdef() interface), that issue disappears. As a side note, ctypes has a large number of users, so even if it were deprecated that wouldn't be a good reason to stop maintaining it. And calling cffi simple while it relies on a parser of the C language (which would then have to be bundled with Python) is a bit misleading IMO. Regards Antoine. ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] libffi embedded in CPython
On 11 March 2015 at 17:27, Brett Cannon br...@python.org wrote: I'm going to propose a somewhat controversial idea: let's deprecate the ctypes module. We now have things like cffi and Cython for people who need to interface with C code. Both of those projects are maintained. And they are not overly difficult to work with. All of this seems to match the reasons we added ctypes in the first place while also being safer to use than ctypes. -1. On Windows a huge amount of code uses ctypes to interface with the Windows API. Many of those projects transitioned from pywin32 because that is a big dependency, and a binary build. Not having in-core access to the Windows API would be a huge problem for many projects. Who would rewrite projects like pyreadline or colorama, for example? How would pip (which has to vendor everything and can't include binary dependencies because of its nature) locate the correct windows config file folders without ctypes in core? Paul ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] libffi embedded in CPython
On Wed, Mar 11, 2015 at 7:50 PM, Antoine Pitrou solip...@pitrou.net wrote: On Wed, 11 Mar 2015 17:27:58 + Brett Cannon br...@python.org wrote: Did anyone ever step forward to do this? I'm a bit worried about the long-term viability of ctypes if we don't have a maintainer or at least someone making sure we are staying up-to-date with upstream libffi. The ctypes module is a dangerous thing, so having a chunk of C code that isn't being properly maintained seems to me to make it even more dangerous. Depends what you call dangerous. C code doesn't rot quicker than pure Python code :-) Also, libffi really offers a wrapper around platform ABIs, which rarely change. And yet, lesser known ABIs in libffi contain bugs (as we discovered trying to work there with anything else than x86 really). Also there *are* ABI differencies that change slowly over time (e.g. requiring stack to be 16 byte aligned) I'm going to propose a somewhat controversial idea: let's deprecate the ctypes module. This is gratuitous. I'm +1 on deprecating ctypes We now have things like cffi and Cython for people who need to interface with C code. Both of those projects are maintained. And they are not overly difficult to work with. But they are not ctypes. For example, cffi wouldn't be obvious to use for interfacing with non-C code, since it requires you to write C-like declarations. You mean like Fortran? Or what precisely? I don't understand why cffi would be safer than ctypes. At least not in the operation mode where it doesn't need to invoke a C compiler. Cython is a completely different beast, it requires a separate compilation pass which makes it useless in some situations. Our main motivation for safer is comes with less magic and less gotchas, which also means does less stuff. It's also smaller. ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] (ctypes example) libffi embedded in CPython
On Wed, 11 Mar 2015 19:05:57 +0100 Antoine Pitrou solip...@pitrou.net wrote: But they are not ctypes. For example, cffi wouldn't be obvious to use for interfacing with non-C code, since it requires you to write C-like declarations. You mean like Fortran? Or what precisely? Any toolchain that can generate native code. It can be Fortran, but it can also be code generated at runtime without there being any external declaration. Having to generate C declarations for such code would be a distraction. For instance, you can look at the compiler example that Eli wrote using llvmlite. It implements a JIT compiler for a toy language. The JIT-compiled function is then declared and called using a simple ctypes declaration: https://github.com/eliben/pykaleidoscope/blob/master/chapter7.py#L937 Regards Antoine. ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Tunning binary insertion sort algorithm in Timsort.
[Tim] 1. Merge 2 at a time instead of just 1. That is, first sort the next 2 elements to be merged (1 compare and a possible swap). Then binary search to find where the smaller belongs, and a shorter binary search to find where the larger belongs. Then shift both into place. [Armin] Good idea, but when I tried that it seemed to increase the total number of comparisons (on random inputs with only up to 136 items). The increase is on the order of 5%. I'm not sure reduced data movement can compensate for that in Python. Which is another way of saying bad idea - that must be why I didn't pursue it to begin with ;-) Thanks for trying! I plugged a similar routine into the code I showed before to count the # of comparisons in Nha Pham's idea, and this merge 2 at a time thing has a higher average # of compares (over all permutations) than Nha's (which in turn has a higher average than the status quo). That makes some sense, thinking about what they do. Nha's algorithm has some supernaturally good cases (input array already ascending or already descending), but merge 2 at a time doesn't appear to have any. In any case, the information-theoretic minimum average number of comparisons for merging N sorted elements with 2 sorted elements is (where do the 2 belong in the final list of N+2 elements? = comb(N+2, 2)): log2((N+2)*(N+1)/2) = log2(N+2) + log2(N+1) - 1 Add a comparison to get the 2 elements in order to begin with, and we're up to log2(N+2) + log2(N+1) Two independent binary inserts (first to a list of size N, and then to a list of size N+1) comes out to the same. So even being supernaturally clever can't reduce the average number of compares this way. And since, in context, we're only looking at short arrays, a marginal saving in data movement costs (which are still O(N**2) worst case) are unlikely to be significant. Still, if anyone wants to go nuts ... ;-) ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] libffi embedded in CPython
On Mar 11, 2015 12:55 PM, Maciej Fijalkowski fij...@gmail.com wrote: On Wed, Mar 11, 2015 at 7:50 PM, Antoine Pitrou solip...@pitrou.net wrote: On Wed, 11 Mar 2015 17:27:58 + Brett Cannon br...@python.org wrote: Did anyone ever step forward to do this? I'm a bit worried about the long-term viability of ctypes if we don't have a maintainer or at least someone making sure we are staying up-to-date with upstream libffi. The ctypes module is a dangerous thing, so having a chunk of C code that isn't being properly maintained seems to me to make it even more dangerous. Depends what you call dangerous. C code doesn't rot quicker than pure Python code :-) Also, libffi really offers a wrapper around platform ABIs, which rarely change. And yet, lesser known ABIs in libffi contain bugs (as we discovered trying to work there with anything else than x86 really). Also there *are* ABI differencies that change slowly over time (e.g. requiring stack to be 16 byte aligned) Are there tests for this? I'm going to propose a somewhat controversial idea: let's deprecate the ctypes module. This is gratuitous. I'm +1 on deprecating ctypes -1. These docs are helpful for understanding the pros and cons of different CPython - C interfaces. https://scipy-lectures.github.io/advanced/interfacing_with_c/interfacing_with_c.html (https://github.com/scipy-lectures/scipy-lecture-notes/issues/131 discusses documenting CFFI here as well) ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] libffi embedded in CPython
On Wed, 11 Mar 2015 17:27:58 + Brett Cannon br...@python.org wrote: Did anyone ever step forward to do this? I'm a bit worried about the long-term viability of ctypes if we don't have a maintainer or at least someone making sure we are staying up-to-date with upstream libffi. The ctypes module is a dangerous thing, so having a chunk of C code that isn't being properly maintained seems to me to make it even more dangerous. Depends what you call dangerous. C code doesn't rot quicker than pure Python code :-) Also, libffi really offers a wrapper around platform ABIs, which rarely change. I'm going to propose a somewhat controversial idea: let's deprecate the ctypes module. This is gratuitous. We now have things like cffi and Cython for people who need to interface with C code. Both of those projects are maintained. And they are not overly difficult to work with. But they are not ctypes. For example, cffi wouldn't be obvious to use for interfacing with non-C code, since it requires you to write C-like declarations. All of this seems to match the reasons we added ctypes in the first place while also being safer to use than ctypes. I don't understand why cffi would be safer than ctypes. At least not in the operation mode where it doesn't need to invoke a C compiler. Cython is a completely different beast, it requires a separate compilation pass which makes it useless in some situations. Regards Antoine. ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 488: elimination of PYO files
On Wed, Mar 11, 2015 at 5:29 PM Armin Rigo ar...@tunes.org wrote: Hi Brett, On 6 March 2015 at 19:11, Brett Cannon br...@python.org wrote: I disagree with your premise that .pyo files don't have a noticeable effect on performance. If you don't use asserts a lot then there is no effect, but if you use them heavily or have them perform expensive calculations then there is an impact. Maybe you'd be interested to learn that PyPy (at least the 2.x branch) uses a new bytecode, JUMP_IF_NOT_DEBUG, to conditionally jump over the assert line. In optimized mode PyPy follows the jumps; in non-optimized mode it doesn't. This mode is initialized with the -O flag but can be changed dynamically, as the bytecode is the same. We introduced it as a simple solution to the mess of .pyc versus .pyo. (We didn't consider the case of -OO very closely because PyPy is much bigger than CPython as a binary to start with, so the demand for that is lower.) Interesting, so you simply merged the optimization levels 0 and 1 in the bytecode and basically drop .pyo files thanks to it. That might be some motivation to support the default file name not having any specified optimization level at all. ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] libffi embedded in CPython
On Wed, Mar 11, 2015 at 6:03 PM Paul Moore p.f.mo...@gmail.com wrote: On 11 March 2015 at 21:45, Maciej Fijalkowski fij...@gmail.com wrote: Is it possible to use cffi without a C compiler/headers as easily than ctypes? yes, it has two modes, one that does that and the other that does extra safety at the cost of a C compiler So if someone were to propose a practical approach to including cffi into the stdlib, *and* assisting the many Windows projects using ctypes for access to the Windows API [1], then there may be a reasonable argument for deprecating ctypes. But nobody seems to be doing that, rather the suggestion appears to be just to deprecate a widely used part of the stdlib offering no migration path :-( You're ignoring that it's not maintained, which is the entire reason I brought this up. No one seems to want to touch the code. Who knows what improvements, bugfixes, etc. exist upstream in libffi that we lack because no one wants to go through and figure it out. If someone would come forward and help maintain it then I have no issue with it sticking around. ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] libffi embedded in CPython
On Wed, 11 Mar 2015 22:20:58 + Brett Cannon br...@python.org wrote: You're ignoring that it's not maintained, According to which metric/criterion? changeset: 94932:86c9ef950288 user:Steve Dower steve.do...@microsoft.com date:Tue Mar 10 09:56:38 2015 -0700 summary: Issue #23606: Disable ctypes.util.find_library(c) on Windows so tests are skipped while we figure out how best to approach the CRT change changeset: 94756:7c6e3358221a user:Serhiy Storchaka storch...@gmail.com date:Thu Feb 26 15:27:57 2015 +0200 summary: Silenced minor GCC warnings. changeset: 94653:a84ae2ccd220 user:Serhiy Storchaka storch...@gmail.com date:Mon Feb 16 20:52:17 2015 +0200 summary: Issue #23450: Fixed possible integer overflows. [etc.] Who knows what improvements, bugfixes, etc. exist upstream in libffi that we lack because no one wants to go through and figure it out. Well, who knows? How about you enlighten us about that? And why do you think cffi, which *also* requires libffi, would be better served in that regard? Regards Antoine. ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] libffi embedded in CPython
On Thu, Mar 12, 2015 at 12:20 AM, Brett Cannon br...@python.org wrote: On Wed, Mar 11, 2015 at 6:03 PM Paul Moore p.f.mo...@gmail.com wrote: On 11 March 2015 at 21:45, Maciej Fijalkowski fij...@gmail.com wrote: Is it possible to use cffi without a C compiler/headers as easily than ctypes? yes, it has two modes, one that does that and the other that does extra safety at the cost of a C compiler So if someone were to propose a practical approach to including cffi into the stdlib, *and* assisting the many Windows projects using ctypes for access to the Windows API [1], then there may be a reasonable argument for deprecating ctypes. But nobody seems to be doing that, rather the suggestion appears to be just to deprecate a widely used part of the stdlib offering no migration path :-( You're ignoring that it's not maintained, which is the entire reason I brought this up. No one seems to want to touch the code. Who knows what improvements, bugfixes, etc. exist upstream in libffi that we lack because no one wants to go through and figure it out. If someone would come forward and help maintain it then I have no issue with it sticking around. It's a bit worse than that. Each time someone wants to touch the code (e.g. push back the upstream libffi fixes), there is we need to review it, but there is noone to do it, noone knows how it works, don't touch it kind of feedback, which leads to disincentives to potential maintainers. I would be likely willing to rip off the libffi from CPython as it is for example (and just use the upstream one) ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] libffi embedded in CPython
On Thu, Mar 12, 2015 at 12:31 AM, Antoine Pitrou solip...@pitrou.net wrote: On Wed, 11 Mar 2015 23:10:14 +0200 Maciej Fijalkowski fij...@gmail.com wrote: Well, sure. The point is, such bugs are unlikely to appear at a fast rate... Also, I don't understand why libffi issues would affect cffi any less than it affects ctypes, at least in the compiler-less mode of operation. My point here was only about shipping own libffi vs using the system one (and it does affect cffi equally with or without compiler) So what? If ctypes used the system libffi as cffi does, it would by construction be at least portable as cffi is. The only reason the bundled libffi was patched at some point was to be *more* portable than vanilla libffi is. So, really, I don't see how switching from ctypes to cffi solves any of this. You're missing my point. Ripping off the libffi from CPython is a good idea to start with. Maybe deprecating ctypes is *also* a good idea, but it's a separate discussion point. It certainly does not solve the libffi problem. ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] libffi embedded in CPython
On Wed, 11 Mar 2015 22:55:33 + Paul Moore p.f.mo...@gmail.com wrote: On 11 March 2015 at 22:33, Maciej Fijalkowski fij...@gmail.com wrote: You're missing my point. Ripping off the libffi from CPython is a good idea to start with. Maybe deprecating ctypes is *also* a good idea, but it's a separate discussion point. It certainly does not solve the libffi problem. OK, so let's focus on the libffi side of things and ignore deprecating or replacing ctypes. I guess I don't see a problem with a proof-of-concept patch to upgrade the libffi (obviously it's not possible to rely on a system libffi on Windows, but treating it as an external might work). If it passes all the tests on all platforms, maybe it could be considered. Agreed. We have enough tests and enough buildbots that it can be attempted. (besides, the only more-or-less officially supported platforms are Linux, Windows, OS X; we also try to not to be too broken on the BSDs, but that's it) Regards Antoine. ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] libffi embedded in CPython
On 11 March 2015 at 22:33, Maciej Fijalkowski fij...@gmail.com wrote: You're missing my point. Ripping off the libffi from CPython is a good idea to start with. Maybe deprecating ctypes is *also* a good idea, but it's a separate discussion point. It certainly does not solve the libffi problem. OK, so let's focus on the libffi side of things and ignore deprecating or replacing ctypes. I guess I don't see a problem with a proof-of-concept patch to upgrade the libffi (obviously it's not possible to rely on a system libffi on Windows, but treating it as an external might work). If it passes all the tests on all platforms, maybe it could be considered. I don't see how anyone can say yes, we'll do it without seeing evidence that it'll work. But equally, I don't think there's any reason to say it absolutely wouldn't be accepted regardless of evidence. So why not prepare a patch for 3.6 (I assume it's too late to make such a big change for 3.5) and we'll see what happens. Or even better, prepare a test build of 3.5 or even 3.4 that switches libffi - people can replace an existing install (I'd be willing to) and test it in live situations. But unless someone provides a patch, the status quo will win. At least until an actual bug that affects live code forces the issue. Paul ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] libffi embedded in CPython
On Wed, 11 Mar 2015 23:10:14 +0200 Maciej Fijalkowski fij...@gmail.com wrote: Well, sure. The point is, such bugs are unlikely to appear at a fast rate... Also, I don't understand why libffi issues would affect cffi any less than it affects ctypes, at least in the compiler-less mode of operation. My point here was only about shipping own libffi vs using the system one (and it does affect cffi equally with or without compiler) So what? If ctypes used the system libffi as cffi does, it would by construction be at least portable as cffi is. The only reason the bundled libffi was patched at some point was to be *more* portable than vanilla libffi is. So, really, I don't see how switching from ctypes to cffi solves any of this. Regards Antoine. ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 488: elimination of PYO files
On Wed, Mar 11, 2015 at 05:34:10PM +, Brett Cannon wrote: I have a poll going on G+ to see what people think of the various proposed file name formats at https://plus.google.com/u/0/+BrettCannon/posts/fZynLNwHWGm . Feel free to vote if you have an opinion. G+ hates my browser and won't let me vote. I click on the button and nothing happens. I have Javascript enabled and I'm not using any ad blockers. For the record, I think only the first two options importlib.cpython-35.opt-0.pyc importlib.cpython-35.opt0.pyc are sane, and I prefer the first. I'm mildly inclined to leave out the opt* part for default, unoptimized code. In other words, the file name holds two or three '.' delimited fields, plus the extension: module.interpreter-version.[opt-optimization code].pyc where [...] is optional and the optimization codes for CPython will be 1 for -O and 2 for -OO. And 0 for unoptimized, if you decide that it should be mandatory. Thank you for moving forward on this, I think it is a good plan. -- Steve ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Why does python use relative instead of absolute path when calling LoadLibrary*
On Wed, Mar 11, 2015 at 4:37 PM David Cournapeau courn...@gmail.com wrote: Hi, While looking at the import code of python for C extensions, I was wondering why we pass a relative path instead of an absolute path to LoadLibraryEx (see bottom for some context). In python 2.7, the full path existence was even checked before calling into LoadLibraryEx ( https://github.com/python/cpython/blob/2.7/Python/dynload_win.c#L189), but it looks like this check was removed in python 3.x branch. I personally don't know of any specific reason. -Brett Is there any defined behaviour that depends on this path to be relative ? Context --- The reason why I am interested in this is the potential use of SetDllDirectory to share dlls between multiple python extensions. Currently, the only solutions I am aware of are: 1. putting the dlls in the PATH 2. bundling the dlls side by side the .pyd 3. patching packages to use preloading (using e.g. ctypes) I am investigating a solution 4, where the dlls would be put in a separate private directory only known of python itself, without the need to modify PATH. Patching python to use SetDllDirectory(some private paths specific to a python interpreter) works perfectly, except that it slightly changes the semantics of LoadLibraryEx not to look for dlls in the current directory. This breaks importing extensions built in place, unless I modify the call in ;https://github.com/python/cpython/blob/2.7/Python/dynload_win.c#L195 from: hDLL = LoadLibraryEx(pathname, NULL LOAD_WITH_ALTERED_SEARCH_PATH) to hDLL = LoadLibraryEx(pathbuf, NULL LOAD_WITH_ALTERED_SEARCH_PATH) That seems to work, but I am quite worried about changing any import semantics by accident. David ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/ brett%40python.org ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Tunning binary insertion sort algorithm in Timsort.
Hi Tim, On 10 March 2015 at 18:22, Tim Peters tim.pet...@gmail.com wrote: 1. Merge 2 at a time instead of just 1. That is, first sort the next 2 elements to be merged (1 compare and a possible swap). Then binary search to find where the smaller belongs, and a shorter binary search to find where the larger belongs. Then shift both into place. Good idea, but when I tried that it seemed to increase the total number of comparisons (on random inputs with only up to 136 items). The increase is on the order of 5%. I'm not sure reduced data movement can compensate for that in Python. Test and code available here: https://bitbucket.org/arigo/arigo/src/default/hack/pypy-hack/list_sort/ The change to insert two items at a time is here: https://bitbucket.org/arigo/arigo/commits/68e04d143dc242cfd9e3934451321f685a68a8e2 (This is taken straight from PyPy's code.) A bientôt, Armin. ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
[Python-Dev] Not documented special methods
There are many special names used in Python core and the stdlib, but absent in the documentation index [1]. If you see names that are defined or used in the module or area maintained by you, please add references to these names to the index and document them if it is needed. Repeat the lists here. Module level names used in pydoc: __author__ __credits__ __date__ __version__ Module level name used in doctest: __test__ Other module level names: __about__ (heapq only) __copyright__ (many modules) __cvsid__ (tarfile only) __docformat__ (doctest only) __email__ (test_with and test_keywordonlyarg only) __libmpdec_version__(decimal only) __status__ (logging only) type attributes (mostly used in tests): __abstractmethods__ (used in abc, functools) __base__ __basicsize__ __dictoffset__ __flags__ (used in inspect, copyreg) __itemsize__ __weakrefoffset__ super() attributes: __self_class__ __thisclass__ Used in sqlite: __adapt__ __conform__ Used in ctypes: __ctype_be__ __ctype_le__ __ctypes_from_outparam__ Used in unittest: __unittest_expecting_failure__ __unittest_skip__ __unittest_skip_why__ float methods, for testing: __getformat__ __setformat__ Used in IDLE RPC: __attributes__ __methods__ Others: __alloc__ (bytearray method) __args__(used in bdb) __build_class__ (builtins function, used in eval loop) __builtins__(module attribute) __decimal_context__ (used in decimal) __exception__ (used in pdb) __getinitargs__ (used in pickle, datetime) __initializing__(used in importlib) __isabstractmethod__(function/method/descriptor attribute, used in abc, functools, types) __ltrace__ (used in eval loop, never set) __members__ (Enum attribute, used in many modules) __mp_main__ (used in multiprocessing) __new_member__ (Enum attribute, used in enum internally) __newobj__ (copyreg function, used in pickle, object.__reduce_ex__) __newobj_ex__ (copyreg function, used in pickle, object.__reduce_ex__) __objclass__(descriptor/enum attribute, used in inspect, pydoc, doctest, multiprocessing) __prepare__ (metaclass method, used in builtins.__build_class__, types) __pycache__ (cache directory name) __return__ (used in pdb) __signature__ (used in inspect, never set) __sizeof__ (standard method, used in sys.getsizeof) __slotnames__ (used in object.__getstate__ for caching) __text_signature__ (function/method/descriptor attribute, used in inspect) __trunc__ (used in math.trunc, int, etc) __warningregistry__ (used in warnings) __weakref__ (used in weakref) __wrapped__ (used in inspect, functools, contextlib, asyncio) [1] http://bugs.python.org/issue23639 ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
