mark florisson, 05.10.2011 15:53:
On 5 October 2011 08:16, Stefan Behnel wrote:
mark florisson, 04.10.2011 23:19:
Another issue is that Cython compile time is increasing with the
addition of control flow and cython utilities. If you use fused types
you're also going to combinatorially add more compile time.
I don't see that locally - a compiled Cython is hugely fast for me. In
comparison, the C compiler literally takes ages to compile the result. An
external shared library may or may not help with both - in particular, it is
not clear to me what makes the C compiler slow. If the compile time is
dominated by the number of inlined functions (which is not unlikely), a
shared library + header file will not make a difference.
Have you tried with the memoryviews merged?
No. I didn't expect the difference to be quite that large.
e.g. if I have this code:
from libc.stdlib cimport malloc
cdef int[:] slice =<int[:10]> <int *> malloc(sizeof(int) * 10)
[0] [14:45] ~ ➤ time cython test.pyx
cython test.pyx 2.61s user 0.08s system 99% cpu 2.695 total
[0] [14:45] ~ ➤ time zsh compile
zsh compile 1.88s user 0.06s system 99% cpu 1.946 total
where 'compile' is the script that invoked the same gcc command
distutils uses. As you can see it took more than 2.5 seconds to
compile this code (simply because the memoryview utilities get
included).
Ok, that hints at serious performance problems. Could you profile it to see
where the issues are? Is it more that the code is loaded from an external
file? Or the fact that more utility code is parsed than necessary?
It's certainly not obvious why the inclusion of static code, even from an
external file, should make any difference.
That being said, it's not we were lacking the infrastructure for making
Python code run faster ...
I'm sure
this came up earlier, but I really think we should have a libcython
and a cython.h. libcython (a shared library) should contain any common
Cython-specific code not meant to be inlined, and cython.h any types,
macros and inline functions etc.
This has a couple of implications though. In order to support this on the
user side, we have to build one shared library per installed package in
order to avoid any Cython versioning issues. Just installing a versioned
"libcython_x.y.z.so" globally isn't enough, especially during development,
but also at deployment time. Different packages may use different CFLAGS or
Cython options, which may have an impact on the result. Encoding all
possible factors in the file name will be cumbersome and may mean that we
still end up with a number of installed Cython libraries that correlates
with the number of installed Cython based packages.
Hm, I think the CFLAGS are important so long as they are compatible
with Python. When the user compiles a Cython extension module with
extra CFLAGS, this doesn't affect libpython. Similarly, the Cython
utilities are really not the user's responsibility, so libcython
doesn't need to be compiled with the same flags as the extension
module. If still wanted, the user could either recompile python with
different CFLAGS (which means libcython will get those as well), or
not use libcython at all. CFLAGS should really only pertain to user
code, not to the Cython library, which the user shouldn't be concerned
about.
Well, it's either the user or the OS distribution that installs (and
potentially builds) the libraries. That already makes it two responsible
entities for many systems that have to agree on what gets installed in what
way. I'm just saying, don't underestimate the details in world wide
deployments.
Stefan
_______________________________________________
cython-devel mailing list
[email protected]
http://mail.python.org/mailman/listinfo/cython-devel
