I have been looking into the LocalJobRunner today. Is there a chance for official support for parallel map execution/>1 reduce tasks or should I look into adding it to my local copy of the code?
Thorsten On 8/19/07, Thorsten Schuett <[EMAIL PROTECTED]> wrote: > > In my case, it looks as if the loopback device is the bottleneck. So > increasing the number of tasks won't help. > > Thorsten > > On 8/18/07, Ted Dunning <[EMAIL PROTECTED]> wrote: > > > > > > > > You might try increasing the number of map and reduce tasks so that you > > can > > overlap cpu and I/O. It is common in parallel applications that you > > need to > > do something like this. > > > > > > On 8/18/07 8:36 AM, "Thorsten Schuett" <[EMAIL PROTECTED] > wrote: > > >> If my assumptions are correct, would it be possible to > > >>> read/access the files directly in the "one-node mode"? > > >> > > >> Please take a look at LocalJobRunner in src/org/apache/hadoop/mapred > > ... > > >> set the jobtracker in your config to 'local' and this happens > > automatically. > > >> (http://wiki.apache.org/lucene-hadoop/HowToDebugMapReducePrograms ) > > > > > > > > > When I use "local", I loose the web interface and the multi-threading. > > I can > > > live with the former, but the latter is not an option. > > > > > > Thorsten > > > > >
