hi gael,
> I am not too entousiastic about that: joblib is meant to be a light
> library. Such a solution would pretty much force the joblib release
> manager to have an SGE cluster in order to run the tests and debug.
>
> Just like I think that IPython-specific stuff should live in IPython, I
> think that cluster-specific stuff should live in their own world. Maybe
> something built on top of libdrmaa? I am incompetent in the field of
> clusters.
>
i understand, but you have to have a glue somewhere unless each of these
distribution libraries expose the same api. the python interface to drmaa
has been improving - i'll take a look again.
> However, if we design right the pool-dispatch mechanism that we are
> discussing for joblib, it should be able to take a pool implemented on
> top of any job dispatching meachnism.
>
the problem is that you have to agree on the data model for job dispatch
and that might take a while before all these places expose a uniform worker
api. don't get me wrong, i would love to see it happen at a library level,
we are not quite there yet. in some sense what soizic has done with soma is
kind of that, but it needs to be a much more open community process similar
to the route multiprocessing took.
cheers,
satra
------------------------------------------------------------------------------
Try before you buy = See our experts in action!
The most comprehensive online learning library for Microsoft developers
is just $99.99! Visual Studio, SharePoint, SQL - plus HTML5, CSS3, MVC3,
Metro Style Apps, more. Free future releases when you subscribe now!
http://p.sf.net/sfu/learndevnow-dev2
_______________________________________________
Scikit-learn-general mailing list
[email protected]
https://lists.sourceforge.net/lists/listinfo/scikit-learn-general
