Hi Jiang, thanks for your response. I think the idea would be to be able to use the map-reduce programming paradigm on small, local jobs. In other words, provide a way to take existing jobs that are running in a distributed fashion and run them against the machine-local version. Part of the purpose is educational, and intended to illustrate the way that map-reduce is implemented and the trade-offs that are present. I hope this clarifies things.
On 11/25/12 9:54 PM, [email protected] wrote: > Hi Randy, > The intermediate key-value pairs are not written to HDFS. They are written to > the local file system. Besides, if the job is "small", why do you use the > MapReduce? You can just do it on a local machine. > > Jiang Shan > > > > > > From: rshepherd > Date: 2012-11-26 09:38 > To: mapreduce-dev > Subject: map-reduce-related school project help > Hi everybody, > > I am a student at NYU and am evaluating an idea for final project for a > distributed systems class. The idea is roughly as follows; the overhead > for running map-reduce on a 'small' job is high. (A small job would be > defined as something fitting in memory on a single machine.) Can > hadoop's map-reduce be modified to be efficient for jobs such as this? > > It seems that one way to do begin to achieve this goal would be to > modify the way the intermediate key-value pairs are handled, the > "handoff" from the map to the reduce. Rather than writing them to HDFS, > either pass them directly to a reducer or keep them in memory in a data > structure. Using a single, shared hashmap would alleviate the need to > sort the mapper output. Instead perhaps distribute the slots to a > reducer or reducers on multiple threads. My hope is that, as this is a > simplification of distributed map-reduce, it will be relatively > straightforward to alter the code to in-memory approach for smaller jobs > that would perform very well for this special case. > > I was hoping that someone on the list could help me with the following > questions: > > 1) Does this sound like a good idea that might be achievable in a few weeks? > 2) Does my intuition about how to achieve the goal seem reasonable? > 3) If so, any advice on now to navigate the code base? (Any pointers on > packages/classes of interest would be highly appreciated) > 4) Any other feedback? > > Thanks in advance to anyone willing and able to help! > Randy
