Patches item #1454844, was opened at 2006-03-20 19:41 Message generated for change (Comment added) made by zpincus You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=305470&aid=1454844&group_id=5470
Please note that this message will contain a full copy of the comment thread, including the initial issue submission, for this request, not just the latest update. Category: Build Group: Python 2.5 Status: Open Resolution: None Priority: 5 Submitted By: Zach Pincus (zpincus) Assigned to: Nobody/Anonymous (nobody) Summary: Use dlopen() to load extensions on Darwin, where possible Initial Comment: Here is a patch to make Python on OS X 10.3 and above use dlopen() (via dynload_shlib.c) to load extension modules, and to make the dl module avaliable. The patch was generated against the SVN head as of yesterday, but can be applied (and I have done so with success) to Python 2.4.2. RATIONALE --------- On most unix-like operating systems, Python uses the dlopen() call to load extension modules. In addition, the way that these modules are opened can be modified via 'sys.setdlopenflags(...)'. Modifications of how extensions are loaded are useful for several reasons (enough so that the standard Python docs (e.g. http:// docs.python.org/lib/module-sys.html ). In particular, if multiple modules need to share symbols, a call to sys.setdlopenflags is necessary. Symbol sharing is especially important for interoperability of modules which wrap C++ classes, because GCC creates classes that resolve their run-time type identification by symbol identity. Thus, symbols must be shared globally for many C++ features to work properly for objects passed between modules. On OS X / Darwin, Python uses some NeXT-derived APIs to load modules. Though these APIs provide analogues to the dlopenflags used to control how dlopen() loads modules, this interface is *not* exposed to the Python interpreter. Worse, sys.setdlopenflags remains available on Darwin, though calls to it are never heeded. Fortunately, on OS X 10.3 and above, Apple has included dlopen as a standard function. In 10.3, this call is provided by a compatibility API; in 10.4, the dlopen() call is written to interface directly with the library loading mechanism and is now the recommended method for libraries to be opened for non Carbon/Cocoa tools. IMPLEMENTATION -------------- This (trivial) patch instructs the Python build process to use dynload_shlib.c (which uses dlopen) instead of dynload_next.c (which uses the NeXT-derived APIs). It also allows for the dl module to be built in order to provide access to the proper values for the various dlopen flags. TESTING ------- This patch can be configured and built into executables that build and test correctly on 10.3 and 10.4. Because Python 2.5 and 2.4 do not currently compile properly on OS X 10.2, I have not built or tested this patch on that OS version. However, the configure and compile process does select the appropriate dynload_next.c file to use, and compiles that correctly before breaking elsewhere. Thus, if the other errors are fixed for 10.2, these patches will work fine. (This is because they only change Python's behavior for 10.3 and up.) PATCHES ------- There are three main components to the attached patch. The first is a patch the 'configure.in' file to use dynload_shlib.c when it can, and a patch to the 'configure' file to sync it up with 'configure.in'. The second is a minor change to 'setup.py' and the dl module test to allow the dl module to be built and tested on OS X systems where dlfcn is available. (10.3 and above.) The last part of the patchfile should be considered optional. This patch applies to 'Lib/test/regrtest.py', and it tells the testing suite that the dl test is not expected to be skipped anymore. This is optional because if Python is ever built on 10.2, the test script will expect dl to work, when it only works on 10.3 and above. However, if Python on 10.2 is officially not supported, then this change should be made to properly test the dl functionality on all supported OS X platforms. ---------------------------------------------------------------------- >Comment By: Zach Pincus (zpincus) Date: 2006-04-03 14:43 Message: Logged In: YES user_id=748718 Another point -- The functions used in dynload_next.c, which is what Python currently uses to load extensions on OS X, are officially 'discouraged' by Apple, in favor of the dlopen() functions: http://developer.apple.com/documentation/DeveloperTools/Reference/ MachOReference/index.html Here's dynload_next.c, for reference. http://svn.python.org/view/python/trunk/Python/dynload_next.c?view=auto Thus, it rather makes sense to move Python from this old and in all probability soon-to-be-deprecated code path. ---------------------------------------------------------------------- Comment By: Zach Pincus (zpincus) Date: 2006-03-27 18:40 Message: Logged In: YES user_id=748718 Ronald - I am unfortunately not an expert on windows DLL loading. I do know that in non-GCC C++ implementations, it becomes far less critical to share symbols globally in many circumstances, because non-GCC C++ implementations tend to use string comparisons to resolve C++ object identity, and not use exact address comparisons on the typeinfo objects. Thus, these objects do not need to be shared between windows DLLs. Beyond that, I do not know whether symbols from DLLs are loaded with global visibility in windows or not. Certainly the windows dlopen() equivalent *does not* support mode options. I presume anything tagged with _declspec (dllexport) is loaded globally when the DLL is opened, but I am not sure. However, that issue is largely tangential to the question of whether Python on OS X should work like Python on every other Unix-like system... As to your other question -- have I checked if this will break existing python code -- the answer is of course yes. The python regression suite test passes on both 10.3 and 10.4 with this modification (as I mentioned), so that's a good indication that run-of-the-mill code will not be affected. In fact, on 10.3, the dlopen() call is implemented under the hood with the exact same calls that Python's dynload_next.c file uses. In 10.4 this was changed so that both APIs talk to the same underlying system. Thus, nothing is really changing: both are functionally identical ways of getting the same OS X runtime loader (dyld) to load a bundle from disk. The only difference is that by using dlopen(), there are hooks in Python for manipulating the mode that is used to load that bundle from disk. So, I cannot imagine a situation where this change would break code on the mac. While argument from failure of imagination is typically weak, this combined with the fact that the python tests pass without difficulty (including the dl module test), I am fairy confident in this patch. Moreover, I have been using this patch on my own computer for some time and have not run into any difficulties, even when loading fairly complex custom extensions. Finally, this patch *only* modifies the codepath on Darwin/OS X, so there's zero possibility that it would affect anything on other platforms. ---------------------------------------------------------------------- Comment By: Ronald Oussoren (ronaldoussoren) Date: 2006-03-27 17:25 Message: Logged In: YES user_id=580910 I have a number of questions. Firstly, how does symbol sharing between extensions work on windows? And more importantly, have you checked if this will break existing python code? ---------------------------------------------------------------------- Comment By: Zach Pincus (zpincus) Date: 2006-03-22 08:34 Message: Logged In: YES user_id=748718 I just uploaded a new patch which addresses this issue, and fixes this same problem in a few other locations in the configure script. ---------------------------------------------------------------------- Comment By: Andrew Barnert (barnert) Date: 2006-03-22 02:31 Message: Logged In: YES user_id=1473180 One minor issue: Your pattern will match Darwin 10.0, which will come out in a few years as part of OS X 10.7, by which time nobody will remember why this script is written this way. Since it's easy to handle now, we might as well do so. On a side note, there are no 2.x-4.x versions (1.5 was renamed 5.1), so anyone who matches Darwin/2.* will be a false match. Here's what I'd suggest: + Darwin/[0156]\..*) DYNLOADFILE="dynload_next.o";; Meanwhile, your patch seems to work for me on an Intel iMac, but I haven't tried anything too sophisticated. ---------------------------------------------------------------------- You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=305470&aid=1454844&group_id=5470 _______________________________________________ Patches mailing list [email protected] http://mail.python.org/mailman/listinfo/patches
