On Fri, Jan 27, 2012 at 05:01:12PM +0100, Andreas wrote:
> I was not sure whether you want other backends in joblib.
The parallel part of joblib is meant to be a convenience wrapper, not a
parallel execution model of its own. As such, enriching it with other
backends is definitely in the scope as long as it doesn't require much
code.
> > Any takers? Their's probably a fair amount of work.
> What do you think would be the hard part / why
> do you think this is much work?
I think that their is a bit of learning to be done. In particular, the
code would have to be well tested, and I have no idea of what the right
way to test it would be. The second problem would be to devise the
extensive test suite, and be convinced that they are no race conditions
when gathering the results. I have found that this is tricky code.
> I have no idea about these things and not much of an idea of how joblib
> works but changing the backends looks not-so-hard from my naive point
> of view.
No, probably not. The 2 main bodies of work would be:
1. To extract out the pool management code from the joblib.Parallel
object to be able to use other kinds of pools than the
multiprocessing pool.
2. To implement a dispatch mechanism to be able to decide which pool the
Parallel object gets.
None of these sound terribly hard, but that's still a bit of effort to
make it work reliably.
In the past, when developping such changes in joblib with a specific
scikit-learn driven usecase, I have done a branch of scikit-learn
replacing the local copy of joblib by an import to the real joblib (you
can look at how it is doen in the debian branch, for instance
https://github.com/scikit-learn/scikit-learn/tree/debian/sklearn/externals
)
Then, I branched out joblib, and made my changes in the joblib branch,
testing them on real-world usecases with the scikit.
That's a way of getting short term results without having to do a full
release cycle of joblib.
Gaël
------------------------------------------------------------------------------
Try before you buy = See our experts in action!
The most comprehensive online learning library for Microsoft developers
is just $99.99! Visual Studio, SharePoint, SQL - plus HTML5, CSS3, MVC3,
Metro Style Apps, more. Free future releases when you subscribe now!
http://p.sf.net/sfu/learndevnow-dev2
_______________________________________________
Scikit-learn-general mailing list
[email protected]
https://lists.sourceforge.net/lists/listinfo/scikit-learn-general
