On 01/27/2012 04:55 PM, Gael Varoquaux wrote: > On Fri, Jan 27, 2012 at 03:44:31PM +0100, Andreas wrote: > >> as it could be. So I was wondering whether there would be a >> non-intrusive way to make sklearn parallelize over the cluster. >> > This is a very legitimate question. Basically, it boils down to: how can > we extend the parallelism model in scikit-learn. > > The way I see it, we would need to define a basic API for parallel > computing that we need. We could start from what we have, that is > parallel maps. > > Parallel maps would make me happy for the moment.
Having parallel maps provided by joblib with a custom backend (first would be IPython) would be perfect. > That way, scikit-learn gets IPython parallelism for free, and can use > multiprocessing as a fallback. > > That's my vision. That vision seems to fulfill all my needs ;) I was not sure whether you want other backends in joblib. Obviously integrating IPython should require as little code as possible. > I lack man-power to develop it. If people are > interested, we can discuss a bit more technical details about how to > implement it. > > Any takers? Their's probably a fair amount of work. > > What do you think would be the hard part / why do you think this is much work? I have no idea about these things and not much of an idea of how joblib works but changing the backends looks not-so-hard from my naive point of view. Having it working and having it in scikit-quality are of course different pairs of shoes. I'd be happy with "working before my deadline" for the moment ;) Cheers, Andy ------------------------------------------------------------------------------ Try before you buy = See our experts in action! The most comprehensive online learning library for Microsoft developers is just $99.99! Visual Studio, SharePoint, SQL - plus HTML5, CSS3, MVC3, Metro Style Apps, more. Free future releases when you subscribe now! http://p.sf.net/sfu/learndevnow-dev2 _______________________________________________ Scikit-learn-general mailing list [email protected] https://lists.sourceforge.net/lists/listinfo/scikit-learn-general
