Good point. I'm no expert in the details of the algorithms per se, but I wonder how the Apache Mahout folks are doing it using map/reduce. Is there a data model in scikit that would be suitable for a map/reduce algorithm approach?
I know ipython can do map/reduce rather easily. On Mon, 2012-05-07 at 20:36 +0200, Andreas Mueller wrote: > On 05/07/2012 08:31 PM, Darren Govoni wrote: > > Thanks Andy. I'll look at it and perhaps try to complete it. I think > > scikit and ipython are natural fit in this age of cloud computing. > I think the main question is at which level you want to parallelize. > In my experience it pays most to parallelize for model selection, > as you have to do that any way and it is so easy to do. > Certain models, such as forests, are also naturally parallel. > If you want to do a gradient descent in parallel using multiple boxes > (or even CPUs), that's a completely different story and probably > quite hard to do right. > > > ------------------------------------------------------------------------------ > Live Security Virtual Conference > Exclusive live event will cover all the ways today's security and > threat landscape has changed and how IT managers can respond. Discussions > will include endpoint security, mobile security and the latest in malware > threats. http://www.accelacomm.com/jaw/sfrnl04242012/114/50122263/ > _______________________________________________ > Scikit-learn-general mailing list > [email protected] > https://lists.sourceforge.net/lists/listinfo/scikit-learn-general > ------------------------------------------------------------------------------ Live Security Virtual Conference Exclusive live event will cover all the ways today's security and threat landscape has changed and how IT managers can respond. Discussions will include endpoint security, mobile security and the latest in malware threats. http://www.accelacomm.com/jaw/sfrnl04242012/114/50122263/ _______________________________________________ Scikit-learn-general mailing list [email protected] https://lists.sourceforge.net/lists/listinfo/scikit-learn-general
