On Oct 05, 2010, at 07:04 PM, P.J. Eby wrote: >At 04:31 PM 10/5/2010 -0400, Barry Warsaw wrote: >>1) Why does setuptools write a stub loader for the .so in the first place? >> Why not just let the import of the shared library happen >> directly using the >> normal Python import rules? > >It does. The normal import rules load .so files first, so the stub >loaders only have an effect if the module is imported from a >*zipfile* (where .so files won't normally load at all). > >When unzipped, the .py file is ignored, and so has no effect. But >when zipped, it forces extraction of the .so to a cache directory, >where it can then be imported.
Interesting, that makes sense, thanks.
I hit this while debugging interpreter crashes during my multi-build work. I
think this assumption is problematic in that case.
Let's say I install extension module 'stupid' using a python3.2m build
(i.e. +pymalloc !wide-unicode !pydebug). This has an _stupid.c extension which
gets built and installed in the egg directory under site-packages, as
_stupid.cpython-32m.so. A _stupid.py stub also gets installed. This hard
codes loading _stupid.cpython-32m.so.
Now I come along and run a python3.2dmu interpreter for which _stupid has not
been installed. In that interpreter I do an import of 'stupid' which imports
_stupid. Now, because the two builds share site-packages (which *should* be
fine since .pyc's don't care about the build flags and .so's are distinguished
by the build flags), python3.2dmu sees the stupid package. It imports
stupid.py which imports _stupid. This triggers the stub, but it tries to load
_stupid.cpython-32m.so (the file name is hardcoded in the stub), and Python
crashes.
I have a patch for distribute (which should be adapted to setuptools) that
essentially substitutes the current interpreter's build flags into
_stupid.{}.so so that the right one will get loaded. This fixes the crashes,
except then I hit the second problem...
>>2) Why does build_ext.py and bdist_egg.py have similar but slightly different
>> stub writers?
>>
>> I'm sure there's an important reason for this, but at least in the case of
>> shared library loading, it seems like the two stubs ought to be more
>> similar than they are.
>
>Dynamic shared library support in setuptools is still officially an
>experimental feature, so there's some possibility that this is a
>bug. That is, bdist_egg's stubs may be broken for extensions using
>dynamic library loader stubs.
'k. Definitely seems like some refactoring is in order.
>>3) Why does site-packages/<package>.egg/ get deleted when the package is
>> re-installed?
>
>Because if you're reinstalling it, it's presumably because the
>original is fux0red in some manner. Also, depending on your
>command-line options, you might be installing an .egg file where a
>directory existed before, or vice versa.
It probably would be better to only blow away the original egg when a
--reinstall option were given. This might change current semantics too much,
so a --keep option would probably be in order.
>> Can this be prevented?
>>
>> This is actually the show-stopper I'm stuck on because if I install my
>> extension with build-A of Python, then try to install it with build-B of
>> Python, the build-A version gets wiped. Because the shared libraries now
>> have build-flag discriminators in the file name, the two installs *should*
>> be able to coexist,
>
>On a version of Python that does this, it would probably be best if the
>platform string included the build flags -- then you would have two separate
>.eggs, each of which would only be loaded by a compatible runtime.
For zip'd eggs, you might be right. For egg directories, it's not necessary.
Everything is sharable, and when not (i.e. the .so's) they are build-flag
discriminated. I'd like to share as much as possible, but I'm open to any
suggestions.
>>Thanks for any feedback you can provide.
>
>Thanks for asking!
Thanks for the explanations! :)
-Barry
signature.asc
Description: PGP signature
_______________________________________________ Distutils-SIG maillist - [email protected] http://mail.python.org/mailman/listinfo/distutils-sig
