coolio! from numba page: "For e.g., llvm-gcc can be used to generate the LLVM assembly for a given C source file, which can then be loaded and manipulated (adding profiling code to every function, say) using a llvm-py based Python script." I wonder if you have tried to profile the beast to see if there is an obvious single bottleneck? ;)
Cheers, On Wed, 07 Aug 2013, Mathieu Blondel wrote: > Hi everyone, > I have been interested in trying my hand at Numba [1] for quite a long > time. For those who don't know, Numba is a just-in-time compiler for > Python and NumPy. When a function is called for the first time, Numba > infers its types and compiles a specialized instance of the function to > LLVM instructions. Numba integrates seamlessly in Python source code: to > accelerate a function, one simply needs to annotate it with a Python > decorator. Therefore, unlike Cython, there is no need for prior > compilation at all. > This evening, I finally gave Numba a shot and reimplemented my sparse > multiclass classification paper [2] using it. The code source is available > in a gist [3]. > My motivation was two-fold: 1) I wanted to compare Numba and Cython in > terms of ease of use and speed for real-world machine learning algorithms > 2) I wanted a more self-contained implementation of my paper since the > reference Cython implementation is quite packed with features and hard to > follow. > I bencharked the Numba implementation against the reference Cython > implementation on the 20 newgroups dataset (18,846 samples; 130,107 > features; 20 classes). The Numba implementation takes 55 seconds to make > 20 outer iterations over the dataset while the Cython implementation takes > 26 seconds. So there is only roughly a 2x performance hit (I'm pretty sure > that both implementations have some room for improvement, though). > I'm quite impressed with Numba and think it has the potential to be a game > changer for the Python scientific community. While reimplementing my > paper, I encountered a few cryptic error messages which make me think it > may be a bit too early to use Numba in scikit-learn yet. However, in a > year or so, after Numba reaches 1.0, I think we should definitely consider > making it a dependency. It should allow us to get rid of much compiled > code in our code base. > Cheers, > Mathieu > [1] [1]http://numba.pydata.org/ > [2] [2]http://www.mblondel.org/code/mlj2013/ > [3] [3]https://gist.github.com/mblondel/6165747 > References > Visible links > 1. http://numba.pydata.org/ > 2. http://www.mblondel.org/code/mlj2013/ > 3. https://gist.github.com/mblondel/6165747 > ------------------------------------------------------------------------------ > Get 100% visibility into Java/.NET code with AppDynamics Lite! > It's a free troubleshooting tool designed for production. > Get down to code-level detail for bottlenecks, with <2% overhead. > Download for free and get started troubleshooting in minutes. > http://pubads.g.doubleclick.net/gampad/clk?id=48897031&iu=/4140/ostg.clktrk > _______________________________________________ > Scikit-learn-general mailing list > Scikit-learn-general@lists.sourceforge.net > https://lists.sourceforge.net/lists/listinfo/scikit-learn-general -- Yaroslav O. Halchenko, Ph.D. http://neuro.debian.net http://www.pymvpa.org http://www.fail2ban.org Senior Research Associate, Psychological and Brain Sciences Dept. Dartmouth College, 419 Moore Hall, Hinman Box 6207, Hanover, NH 03755 Phone: +1 (603) 646-9834 Fax: +1 (603) 646-1419 WWW: http://www.linkedin.com/in/yarik ------------------------------------------------------------------------------ Get 100% visibility into Java/.NET code with AppDynamics Lite! It's a free troubleshooting tool designed for production. Get down to code-level detail for bottlenecks, with <2% overhead. Download for free and get started troubleshooting in minutes. http://pubads.g.doubleclick.net/gampad/clk?id=48897031&iu=/4140/ostg.clktrk _______________________________________________ Scikit-learn-general mailing list Scikit-learn-general@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/scikit-learn-general
