It's not clear to me what you're doing, but it all sounds too complicated to me. I have Cython extensions that depend on C++ code, and I don't have to resort to these kinds of contortions.
One model that works well is to build your C code as a shared library. Make each Cython module in a single file, and have it depend on your C library. In my case, I don't use any of the Python/Cython build systems; I just call cythonize to get C++ code, and then build that C++ code as I would build any other C++ code (in CMake). There's then a bit of CMake/Python stuff needed to get CMake to install in the correct Python locations. Another way would be to build your supporting C code as a separate package (with CMake or Autotools, etc.) --- and then use the Python build system for the actual Cython extension. Now you have two packages instead of one, but that doesn't bother me much. I chose the all-CMake way because I put a lot of time and effort into learning CMake, and didn't want to have to put more time and effort into learning another build system, just so I could build Python. But either way should work. -- Elizabeth On Fri, Jun 10, 2016 at 4:18 PM, Nathan Goldbaum <nathan12...@gmail.com> wrote: > Hmm, so I've looked into this a bit further, and it looks like the > metadata isn't going to be useful for us. Many of our extensions can't be > autogenerated by cython because they have non-trivial dependencies or > depend directly on C code. For these extensions, cythonize must be passed > an instantiated Extension object, and the metadata embedded in the compiled > C file just uses whatever metadata is attached to the Extension object. > > I think my original idea - to parse pyx files for cimports and add a test > - should allow us to manage this more safely in the future while still > maintaining fine-grained control over dependencies of C extensions in our > setup.py file. > > That said, I'd love to hear alternate ideas. > > On Fri, Jun 10, 2016 at 1:49 PM, Robert Bradshaw <rober...@gmail.com> > wrote: > >> We write metadata in the generated C files for just this reason. You >> can fall back to something like >> https://gist.github.com/robertwb/25ab9838cc2b9b21eed646834cf4a108 if >> cython is not available. >> >> >> On Fri, Jun 10, 2016 at 10:55 AM, Nathan Goldbaum <nathan12...@gmail.com> >> wrote: >> > The reason we haven't done that is we would like our setup.py script to >> be >> > runnable without cython being installed. I think cythonize is being >> invoked >> > (or something similar to it) by setuptools, using a feature added in >> > setuptools 18.0: >> > https://setuptools.readthedocs.io/en/latest/history.html#id60 >> > >> > Is there a way to use cythonize for this build workflow without >> importing it >> > at the top-level in our setup.py file? >> > >> > FWIW, our setup.py file is here: >> > >> https://bitbucket.org/yt_analysis/yt/src/yt/setup.py?at=yt&fileviewer=file-view-default >> > >> > On Fri, Jun 10, 2016 at 12:49 PM, Robert Bradshaw <rober...@gmail.com> >> > wrote: >> >> >> >> You should be using cythonize rather than listing and maintaining the >> >> Extension definitions themselves. >> >> >> >> >> >> >> http://docs.cython.org/src/quickstart/build.html#building-a-cython-module-using-distutils >> >> >> https://github.com/cython/cython/wiki/enhancements-distutils_preprocessing >> >> >> >> On Fri, Jun 10, 2016 at 9:18 AM, Nathan Goldbaum < >> nathan12...@gmail.com> >> >> wrote: >> >> > Hi all, >> >> > >> >> > I'm working on a pretty large python/cython codebase (yt) that has >> many >> >> > interdependent C extensions written in cython. >> >> > >> >> > I've found it to be pretty hit or miss to depend on contributors to >> >> > manually >> >> > update cython dependency information in our setup.py file. The >> >> > dependency >> >> > information seems to only be used by setuptools to trigger >> >> > recompilation, >> >> > but critically setuptools will happily compile a C extension for the >> >> > first >> >> > time if the depends information is incomplete. This means that during >> >> > development, if I update a cython routine, there's no way to ensure >> that >> >> > other cython routines that cimport the one I modified will be >> recompiled >> >> > unless I manually ensure the depends information is updated whenever >> >> > cython >> >> > code gains or loses a cimport. >> >> > >> >> > To make that more concrete, here's a pull request I just made to yt >> that >> >> > adds missing dependencies for a cython header. Without this pull >> >> > request, >> >> > setuptools fails to recompile these routines when >> selection_routines.pxd >> >> > changes, causing a build failure. >> >> > >> >> > https://bitbucket.org/yt_analysis/yt/pull-requests/2220 >> >> > >> >> > I think it should be possible to write a test for this by defining >> the >> >> > dependency information outside of setup.py and parsing grep and >> looking >> >> > for >> >> > all cython files that cimport other cython files defined inside yt. >> >> > However, >> >> > before I do that, I'm curious whether anyone has done something >> similar, >> >> > or >> >> > if there is some other way of forcing the dependency information to >> be >> >> > complete on the first compilation, rather than just for subsequent >> >> > incremental recompilations during development. >> >> > >> >> > Thanks for your help! >> >> > >> >> > -Nathan >> >> > >> >> > _______________________________________________ >> >> > cython-devel mailing list >> >> > cython-devel@python.org >> >> > https://mail.python.org/mailman/listinfo/cython-devel >> >> > >> >> _______________________________________________ >> >> cython-devel mailing list >> >> cython-devel@python.org >> >> https://mail.python.org/mailman/listinfo/cython-devel >> > >> > >> > >> > _______________________________________________ >> > cython-devel mailing list >> > cython-devel@python.org >> > https://mail.python.org/mailman/listinfo/cython-devel >> > >> _______________________________________________ >> cython-devel mailing list >> cython-devel@python.org >> https://mail.python.org/mailman/listinfo/cython-devel >> > > > _______________________________________________ > cython-devel mailing list > cython-devel@python.org > https://mail.python.org/mailman/listinfo/cython-devel > >
_______________________________________________ cython-devel mailing list cython-devel@python.org https://mail.python.org/mailman/listinfo/cython-devel
