Re: building c extensions with setuptools that are not tied to python installed on build machine

[Matthew Taylor]

Ned, thank you for your insight on this problem. I will take your
advice and do some more digging. You've been very helpful.

Regards,

-
Matt Taylor
OS Community Flag-Bearer
Numenta


On Wed, Feb 11, 2015 at 4:23 PM, Ned Deily n...@acm.org wrote:
 In article
 cajv6ndphginodqq1fkh1-ubcyzq2chag7qjxsbq0_pht5z8...@mail.gmail.com,
  Matthew Taylor m...@numenta.org wrote:
 Does this make sense to anyone? I'm still a little new to Python in
 general (especially binary packaging), and it seems like this would be
 a common problem for any projects with C extensions that need broad
 binary distribution. Does anyone have any suggestions? Please take a
 look at our setup.py file [4] and tell me if we're doing something
 egregiously wrong.

 Just taking a quick look at your setup.py there shows a quite
 complicated build, including SWIG.  One suggestion: keep in mind that
 it's normal on OS X for the absolute path of shared libraries and
 frameworks to be embedded in the linked binaries, including the C (or
 C++) extension module bundles (.so files) built for Python packages.  If
 any of the .so files or any other binary artifacts (executables, shared
 libraries) created by your package are linked to the Python
 interpreter's shared library, that may be why you are getting a mixture
 of Python instances.  One way to check for this is to use:

 otool -L *.so *.dylib

 on all of the directories containing linked binaries in your project and
 check for paths like:
 /System/Library/Frameworks/Python.framework

 That would be a link to the Apple-supplied system Python.

 A link to /Library/Frameworks/Python.framework or some other path would
 be to a third-party Python like from python.org or Homebrew.

 Due to differences in how the various Pythons are built and depending on
 what the C or C++ code is doing, it may not be possible to have one
 binary wheel that works with different Python instances of the same
 version.  For many simple projects, it does work.

 You *could* also ask on the PythonMac SIG list.

 https://mail.python.org/mailman/listinfo/pythonmac-sig

 --
  Ned Deily,
  n...@acm.org

 --
 https://mail.python.org/mailman/listinfo/python-list
-- 
https://mail.python.org/mailman/listinfo/python-list

building c extensions with setuptools that are not tied to python installed on build machine

[Matthew Taylor]

Hello pythonistas,

This is my first message to this mailing list, so if my question is
off-topic, please direct me to the correct mailing list.

I manage the NuPIC [1] open source machine intelligence project, which
is a python project with C extensions. We have a build on Travis-CI
[2] that creates binary wheels on release tags (in git). I'm having a
problem with OS X binaries.

The Travis-CI build machine is OS X 10.9 and uses the default Python
2.7.6 that comes with OS X. When binaries are created on this machine,
they seem to be linked somehow to the default Python, because when a
user running a homebrew-installed python (or Anaconda) attempts to
install the binary, there's an awful problem [3], even when the Python
versions are exactly the same (2.7.6). It seems that some connection
is made at compile time when C extensions are built that link the
binary file to the system Python on the OS X machine building the
binary. This means any users working with the homebrew Python or
Anaconda will get nasty thread errors when the program attempts to run
against two Pythons.

Fatal Python error: PyThreadState_Get: no current thread

Does this make sense to anyone? I'm still a little new to Python in
general (especially binary packaging), and it seems like this would be
a common problem for any projects with C extensions that need broad
binary distribution. Does anyone have any suggestions? Please take a
look at our setup.py file [4] and tell me if we're doing something
egregiously wrong.

[1] https://github.com/numenta/nupic
[2] https://travis-ci.org/numenta/nupic
[3] https://github.com/numenta/nupic/issues/1813
[4] https://github.com/numenta/nupic/blob/master/setup.py

Thanks thanks in advance,
-
Matt Taylor
OS Community Flag-Bearer
Numenta
-- 
https://mail.python.org/mailman/listinfo/python-list