2012/1/27 Andreas <[email protected]>: > On 01/27/2012 04:55 PM, Gael Varoquaux wrote: >> On Fri, Jan 27, 2012 at 03:44:31PM +0100, Andreas wrote: >> >>> as it could be. So I was wondering whether there would be a >>> non-intrusive way to make sklearn parallelize over the cluster. >>> >> This is a very legitimate question. Basically, it boils down to: how can >> we extend the parallelism model in scikit-learn. >> >> The way I see it, we would need to define a basic API for parallel >> computing that we need. We could start from what we have, that is >> parallel maps. >> >> > Parallel maps would make me happy for the moment. > > Having parallel maps provided by joblib with a custom backend > (first would be IPython) would be perfect. >> That way, scikit-learn gets IPython parallelism for free, and can use >> multiprocessing as a fallback. >> >> That's my vision. > That vision seems to fulfill all my needs ;) > I was not sure whether you want other backends in joblib. > Obviously integrating IPython should require as little code > as possible.
The problem is that for multiprocessing, a n_jobs argument is enough (to tell the number of cores). But for cluster computing you will have to pass some kind of active cluster session (e.g. a IPython.parallel.Client instance that is aware of the configuration of the engines running on the cluster). Maybe it's just a matter or renaming "n_jobs" by something more generic such as "workers" and make joblib smart enough to introspect this guy with some ducktyping to address the parallel case and fallback on it's on multiprocessing backend if the workers object is not an IPython Client or View instance. -- Olivier http://twitter.com/ogrisel - http://github.com/ogrisel ------------------------------------------------------------------------------ Try before you buy = See our experts in action! The most comprehensive online learning library for Microsoft developers is just $99.99! Visual Studio, SharePoint, SQL - plus HTML5, CSS3, MVC3, Metro Style Apps, more. Free future releases when you subscribe now! http://p.sf.net/sfu/learndevnow-dev2 _______________________________________________ Scikit-learn-general mailing list [email protected] https://lists.sourceforge.net/lists/listinfo/scikit-learn-general
