On Feb 3, 2010, at 8:16 AM, David Cournapeau wrote: > On Thu, Feb 4, 2010 at 12:30 AM, Stefan Behnel <[email protected]> > wrote: >> >> David Cournapeau, 03.02.2010 14:45: >>> On Wed, Feb 3, 2010 at 9:11 PM, Stefan Behnel wrote: >>>> Dag Sverre Seljebotn, 03.02.2010 12:20: >>>>> We should have an option to split Cython-generated code into >>>>> reusable >>>>> include-files at some point too, and perhaps make use of >>>>> precompiled >>>>> headers. >>>> Given that the largest part of code in a Cython generated C file >>>> is due to >>>> real user code, I doubt that there is much Cython can improve by >>>> spitting >>>> out multiple files here. >>> >>> I don't think that's always true. Cython quickly generates files >>> which >>> are about 1 Mb of content. For relatively simple code (with a couple >>> of python-visible functions), generated files are 5000 LOC, >>> whereas a >>> hand-made file would be much smaller. >> >> What I meant, was, that the amount of code and the compiler runtime >> is not >> due to code that Cython could extract into header files. Cython >> generates a >> lot of (type) special casing code that you'd never write manually, >> e.g. for >> loops or function argument extraction. The result is that your hand- >> written >> C-API-using code simply isn't as fast as the code that Cython >> generates >> (except for selected cases where Cython isn't good enough yet). >> >> I would also guess that the compile time is substantially increased >> by the >> amount of inline functions that Cython uses for type conversions. > > That was one of my first guess last time I looked at the problem, but > I don't think that's the real culprit : I tried simple things like > removing every inline, and it did not make a difference (both for -O0 > and -O2). > >> The C compiler would still have to handle it, >> regardless of where it came from. > > Not if it is pre-compiled. > >> Parsing is only a tiny part of what the C >> compiler does these days. Optimisation takes several times longer. > > I don't think parsing takes time either. But I have generally observed > that for a unit of compilation of N lines taking T time, 2 * N lines > takes more, sometimes much more than 2 * T, especially for N >= few > thousand lines.
I've heard (somewhat jokingly, but there seems to be some truth to it) that compilation is linear, optimization quadratic in the number of lines. >> When I pass -O3, however, the compile time >> jumps up to some 40 seconds, i.e. more than a factor of 4, which is >> quite >> noticeable in comparison. > > Yes, one of the main advantage of numscons for numpy (and even more > for scipy) is that I can control CFLAGS easily and reliably when > working on it. It is almost one order of magnitude faster than with > distutils if you care about reliability (which often means removing a > lot of built files with distutils). > >> BTW, I know that this is not directly comparable to other projects >> that use >> several separate modules. I guess there is more overhead if you >> have a >> larger number of small Cython modules. But that should be mitigated >> by >> running partial builds (which you'd certainly do during a compile- >> test cycle). There is a huge amount of overhead in something like Sage, at least in terms of space. For example, the (longish) definition of the Element, Parent, SageObject, etc. classes are copied 100s of times. This is an interesting discussion. A bit ago I wrote up http://wiki.cython.org/enhancements/distutils_preprocessing which could use some comments and building upon. I think it's a bit mis-titled, as the application goes beyond disutils (though that could be an orthogonal step). - Robert _______________________________________________ Cython-dev mailing list [email protected] http://codespeak.net/mailman/listinfo/cython-dev
