In short, this is the method google use for parallel processing. Every operation which needs to be parallel is divided into a mapping stage (where each worker does something on their own data, and produces a result) and a reduction stage, where the results of the map are collected into a meaningful result. This parallelization scheme is highly scalable.
This (upcoming) Sunday's slides on Map-Reduce from the concurrent and Distributed Programming course: http://webcourse.cs.technion.ac.il/236370/Winter2008-2009/ho/WCFiles/map-reduce-lecture.pdf And a canonical paper, linked from the same place: http://labs.google.com/papers/mapreduce-osdi04.pdf Orna. On Wed, Jan 14, 2009 at 10:27 PM, guy keren <[email protected]> wrote: > > can you explain, briefly, what map-reduce is, so those not in the know > (like me) will be able to decide if this is interesting? ;) > > thanks, > --guy > > Eran Sandler wrote: > > Hi all, > > > > It's been a while since I've posted to Haifux (or Linux-IL for that > > matter) but I am watching the mailing list from time to time and due to > > personal reasons found myself as a Haifa citizen for the past year (and > > probably for a couple more years :-) ). > > > > Somewhere in 2004 I even did a lecture on Mono, the open source .NET > > implementation, if some of you recall. > > > > Recently I've been involved with a cool open source project called Disco. > > > > Disco is an open source Map-Reduce framework written in Erlang and > > Python. It was written at Nokia's Palo Alto research center as a > > lightweight framework for rapid scripting of distributed data processing > > tasks but grew to become even more than that and is now even used for > > probabilistic modeling, data mining, full text indexing, etc. > > > > You can read more about Disco at http://discoproject.org > > > > Would a lecture on Map-Reduce in general and specifically Disco would > > interest people? > > > > If so, I'm more than willing to give the lecture and show some examples. > > > > Eran > > > > > > ------------------------------------------------------------------------ > > > > _______________________________________________ > > Haifux mailing list > > [email protected] > > http://hamakor.org.il/cgi-bin/mailman/listinfo/haifux > > _______________________________________________ > Haifux mailing list > [email protected] > http://hamakor.org.il/cgi-bin/mailman/listinfo/haifux >
_______________________________________________ Haifux mailing list [email protected] http://hamakor.org.il/cgi-bin/mailman/listinfo/haifux
