Hmm, so I've looked into this a bit further, and it looks like the metadata isn't going to be useful for us. Many of our extensions can't be autogenerated by cython because they have non-trivial dependencies or depend directly on C code. For these extensions, cythonize must be passed an instantiated Extension object, and the metadata embedded in the compiled C file just uses whatever metadata is attached to the Extension object.
I think my original idea - to parse pyx files for cimports and add a test - should allow us to manage this more safely in the future while still maintaining fine-grained control over dependencies of C extensions in our setup.py file. That said, I'd love to hear alternate ideas. On Fri, Jun 10, 2016 at 1:49 PM, Robert Bradshaw <rober...@gmail.com> wrote: > We write metadata in the generated C files for just this reason. You > can fall back to something like > https://gist.github.com/robertwb/25ab9838cc2b9b21eed646834cf4a108 if > cython is not available. > > > On Fri, Jun 10, 2016 at 10:55 AM, Nathan Goldbaum <nathan12...@gmail.com> > wrote: > > The reason we haven't done that is we would like our setup.py script to > be > > runnable without cython being installed. I think cythonize is being > invoked > > (or something similar to it) by setuptools, using a feature added in > > setuptools 18.0: > > https://setuptools.readthedocs.io/en/latest/history.html#id60 > > > > Is there a way to use cythonize for this build workflow without > importing it > > at the top-level in our setup.py file? > > > > FWIW, our setup.py file is here: > > > https://bitbucket.org/yt_analysis/yt/src/yt/setup.py?at=yt&fileviewer=file-view-default > > > > On Fri, Jun 10, 2016 at 12:49 PM, Robert Bradshaw <rober...@gmail.com> > > wrote: > >> > >> You should be using cythonize rather than listing and maintaining the > >> Extension definitions themselves. > >> > >> > >> > http://docs.cython.org/src/quickstart/build.html#building-a-cython-module-using-distutils > >> > https://github.com/cython/cython/wiki/enhancements-distutils_preprocessing > >> > >> On Fri, Jun 10, 2016 at 9:18 AM, Nathan Goldbaum <nathan12...@gmail.com > > > >> wrote: > >> > Hi all, > >> > > >> > I'm working on a pretty large python/cython codebase (yt) that has > many > >> > interdependent C extensions written in cython. > >> > > >> > I've found it to be pretty hit or miss to depend on contributors to > >> > manually > >> > update cython dependency information in our setup.py file. The > >> > dependency > >> > information seems to only be used by setuptools to trigger > >> > recompilation, > >> > but critically setuptools will happily compile a C extension for the > >> > first > >> > time if the depends information is incomplete. This means that during > >> > development, if I update a cython routine, there's no way to ensure > that > >> > other cython routines that cimport the one I modified will be > recompiled > >> > unless I manually ensure the depends information is updated whenever > >> > cython > >> > code gains or loses a cimport. > >> > > >> > To make that more concrete, here's a pull request I just made to yt > that > >> > adds missing dependencies for a cython header. Without this pull > >> > request, > >> > setuptools fails to recompile these routines when > selection_routines.pxd > >> > changes, causing a build failure. > >> > > >> > https://bitbucket.org/yt_analysis/yt/pull-requests/2220 > >> > > >> > I think it should be possible to write a test for this by defining the > >> > dependency information outside of setup.py and parsing grep and > looking > >> > for > >> > all cython files that cimport other cython files defined inside yt. > >> > However, > >> > before I do that, I'm curious whether anyone has done something > similar, > >> > or > >> > if there is some other way of forcing the dependency information to be > >> > complete on the first compilation, rather than just for subsequent > >> > incremental recompilations during development. > >> > > >> > Thanks for your help! > >> > > >> > -Nathan > >> > > >> > _______________________________________________ > >> > cython-devel mailing list > >> > cython-devel@python.org > >> > https://mail.python.org/mailman/listinfo/cython-devel > >> > > >> _______________________________________________ > >> cython-devel mailing list > >> cython-devel@python.org > >> https://mail.python.org/mailman/listinfo/cython-devel > > > > > > > > _______________________________________________ > > cython-devel mailing list > > cython-devel@python.org > > https://mail.python.org/mailman/listinfo/cython-devel > > > _______________________________________________ > cython-devel mailing list > cython-devel@python.org > https://mail.python.org/mailman/listinfo/cython-devel >
_______________________________________________ cython-devel mailing list cython-devel@python.org https://mail.python.org/mailman/listinfo/cython-devel
