2011/10/7 mark florisson <[email protected]>: > On 6 October 2011 21:56, Vitja Makarov <[email protected]> wrote: >> 2011/10/6 mark florisson <[email protected]>: >>> On 6 October 2011 07:46, Stefan Behnel <[email protected]> wrote: >>>> mark florisson, 05.10.2011 15:53: >>>>> >>>>> On 5 October 2011 08:16, Stefan Behnel wrote: >>>>>> >>>>>> mark florisson, 04.10.2011 23:19: >>>>>>> >>>>>>> Another issue is that Cython compile time is increasing with the >>>>>>> addition of control flow and cython utilities. If you use fused types >>>>>>> you're also going to combinatorially add more compile time. >>>>>> >>>>>> I don't see that locally - a compiled Cython is hugely fast for me. In >>>>>> comparison, the C compiler literally takes ages to compile the result. An >>>>>> external shared library may or may not help with both - in particular, it >>>>>> is >>>>>> not clear to me what makes the C compiler slow. If the compile time is >>>>>> dominated by the number of inlined functions (which is not unlikely), a >>>>>> shared library + header file will not make a difference. >>>>> >>>>> Have you tried with the memoryviews merged? >>>> >>>> No. I didn't expect the difference to be quite that large. >>>> >>>> >>>>> e.g. if I have this code: >>>>> >>>>> from libc.stdlib cimport malloc >>>>> cdef int[:] slice =<int[:10]> <int *> malloc(sizeof(int) * 10) >>>>> >>>>> [0] [14:45] ~ ➤ time cython test.pyx >>>>> cython test.pyx 2.61s user 0.08s system 99% cpu 2.695 total >>>>> [0] [14:45] ~ ➤ time zsh compile >>>>> zsh compile 1.88s user 0.06s system 99% cpu 1.946 total >>>>> >>>>> where 'compile' is the script that invoked the same gcc command >>>>> distutils uses. As you can see it took more than 2.5 seconds to >>>>> compile this code (simply because the memoryview utilities get >>>>> included). >>>> >>>> Ok, that hints at serious performance problems. Could you profile it to see >>>> where the issues are? Is it more that the code is loaded from an external >>>> file? Or the fact that more utility code is parsed than necessary? >>> >>> I haven't profiled it yet (I'll do that), but I'm fairly sure it's the >>> parsing of Cython utility files (not the loading). Maybe Tempita also >>> adds to the overhead, I'll find out. >>> >> >> Compiling this regex gives 5ms instead of 10ms on my machine >> >> https://github.com/cython/cython/blob/master/Cython/Compiler/Code.py#L85 >> >> And on your example gives 3% speedup >> > > Sorry, which code gets you 10ms? Also, is this about loading + regex > matching, or just about compiling the pattern? >
I've added decorator for load_utilities_from_file that prints time for current call and total sum for this function and total gives 10ms. Btw that's not that much. > In any case, libcython would solve these issues. Profiling will still > be useful though. > >>>> It's certainly not obvious why the inclusion of static code, even from an >>>> external file, should make any difference. >>>> >>>> That being said, it's not we were lacking the infrastructure for making >>>> Python code run faster ... >>>> >>> >>> Heh, indeed. In this case I think caching will solve all our problems. >>> >>>>>>> I'm sure >>>>>>> this came up earlier, but I really think we should have a libcython >>>>>>> and a cython.h. libcython (a shared library) should contain any common >>>>>>> Cython-specific code not meant to be inlined, and cython.h any types, >>>>>>> macros and inline functions etc. >>>>>> >>>>>> This has a couple of implications though. In order to support this on the >>>>>> user side, we have to build one shared library per installed package in >>>>>> order to avoid any Cython versioning issues. Just installing a versioned >>>>>> "libcython_x.y.z.so" globally isn't enough, especially during >>>>>> development, >>>>>> but also at deployment time. Different packages may use different CFLAGS >>>>>> or >>>>>> Cython options, which may have an impact on the result. Encoding all >>>>>> possible factors in the file name will be cumbersome and may mean that we >>>>>> still end up with a number of installed Cython libraries that correlates >>>>>> with the number of installed Cython based packages. >>>>> >>>>> Hm, I think the CFLAGS are important so long as they are compatible >>>>> with Python. When the user compiles a Cython extension module with >>>>> extra CFLAGS, this doesn't affect libpython. Similarly, the Cython >>>>> utilities are really not the user's responsibility, so libcython >>>>> doesn't need to be compiled with the same flags as the extension >>>>> module. If still wanted, the user could either recompile python with >>>>> different CFLAGS (which means libcython will get those as well), or >>>>> not use libcython at all. CFLAGS should really only pertain to user >>>>> code, not to the Cython library, which the user shouldn't be concerned >>>>> about. >>>> >>>> Well, it's either the user or the OS distribution that installs (and >>>> potentially builds) the libraries. That already makes it two responsible >>>> entities for many systems that have to agree on what gets installed in what >>>> way. I'm just saying, don't underestimate the details in world wide >>>> deployments. >>>> >> >> >> >> -- >> vitja. >> _______________________________________________ >> cython-devel mailing list >> [email protected] >> http://mail.python.org/mailman/listinfo/cython-devel >> > _______________________________________________ > cython-devel mailing list > [email protected] > http://mail.python.org/mailman/listinfo/cython-devel > -- vitja. _______________________________________________ cython-devel mailing list [email protected] http://mail.python.org/mailman/listinfo/cython-devel
