On Fri, Jan 27, 2012 at 11:15:43AM -0500, Satrajit Ghosh wrote: > one option we could consider is to take the job distribution capability in > nipype and make it general purpose and add it to joblib (will require some > effort - it won't be quick). it would be nice as i have stated before to > have joblib be the job distribution guy!
I am not too entousiastic about that: joblib is meant to be a light library. Such a solution would pretty much force the joblib release manager to have an SGE cluster in order to run the tests and debug. Just like I think that IPython-specific stuff should live in IPython, I think that cluster-specific stuff should live in their own world. Maybe something built on top of libdrmaa? I am incompetent in the field of clusters. However, if we design right the pool-dispatch mechanism that we are discussing for joblib, it should be able to take a pool implemented on top of any job dispatching meachnism. Gael ------------------------------------------------------------------------------ Try before you buy = See our experts in action! The most comprehensive online learning library for Microsoft developers is just $99.99! Visual Studio, SharePoint, SQL - plus HTML5, CSS3, MVC3, Metro Style Apps, more. Free future releases when you subscribe now! http://p.sf.net/sfu/learndevnow-dev2 _______________________________________________ Scikit-learn-general mailing list [email protected] https://lists.sourceforge.net/lists/listinfo/scikit-learn-general
