2012/2/1 Gael Varoquaux <[email protected]>: > On Wed, Feb 01, 2012 at 03:05:49PM +0100, Andreas wrote: >> I started working with IPython.parallel for training the trees using joblib. >> It works in principal, but it is SLOW. >> The time between starting and the jobs arriving at the engines is really >> long. >> I'm sending around 20.000x2000 float64 matrices, but this is gigabit >> ethernet and I wouldn't >> expect it to take like 10-20 seconds (haven't measured exactly). > > IPython uses pickling, which is really slow.
This is not the case for plain numpy arrays http://ipython.org/ipython-doc/stable/parallel/parallel_details.html#non-copying-sends-and-numpy-arrays It should also possible to avoid copying the arrays before sending if they are not modified in //. I don't know if this is the case by default or not. -- Olivier http://twitter.com/ogrisel - http://github.com/ogrisel ------------------------------------------------------------------------------ Keep Your Developer Skills Current with LearnDevNow! The most comprehensive online learning library for Microsoft developers is just $99.99! Visual Studio, SharePoint, SQL - plus HTML5, CSS3, MVC3, Metro Style Apps, more. Free future releases when you subscribe now! http://p.sf.net/sfu/learndevnow-d2d _______________________________________________ Scikit-learn-general mailing list [email protected] https://lists.sourceforge.net/lists/listinfo/scikit-learn-general
