Hey, For this year's summer of code, and for my master dissertation, we created a project to compile array expressions efficiently (Dag was my mentor, CCed), which can be found here: https://github.com/markflorisson88/minivect , the thesis is under subdirectory 'thesis'. It's currently integrated in Cython, and can significantly outperform open source and commercial Fortran compilers for certain benchmarks (see the bench/graphs subdirectory on github, or the dissertation). For other benchmarks, it's mostly close, accounting for some runtime overhead.
The purpose of the project is to be reusable, it's used by Cython, it's likely going to be integrated into Theano, it's partially integrated into numba at the moment. It can generate C code for static compilers, and uses llvmpy to do just-in-time specialization for runtime compilers (likely combined with some form of lazy evaluation). However, I think pypy has its own approach, and I think it relies on the JIT to evaluate the expressions (feel free to point me to documentation or source code)? But I don't imagine it optimizes for the cache hierarchy? So the question is, would there be an interest to use this project in PyPy? It's currently meant to be used as a git submodule, or to be included verbatim, but I also intend to make it distributable and installable. The project is designed so that adding new code generators should require minimal effort. But in the event this type of code translation is not compatible with pypy's approach, you could have a look at the techniques for optimizations, such as tiling for spatial locality and several of the optimizations described in the thesis and prototyped in code on github, such as SIMD transposes when mixing C and Fortran contiguous arrays, a tractable way to prove data independence for NumPy arrays to avoid array temporaries, optimizing broadcasting through loop-invariant code motion, and so forth. Anyway, it doesn't have all functionality yet, for instance it doesn't support reductions yet. But let us know if pypy would want to use it, and if so, how we could collaborate and make minivect compatible for the purposes of pypy. Mark _______________________________________________ pypy-dev mailing list pypy-dev@python.org http://mail.python.org/mailman/listinfo/pypy-dev
