Hi Gaurav, I do need parallel processing for the part that I am doing in the mapper right now, and then the results are aggregated. That part of the problem is embarrassingly parallel and data is so huge it can't be processed on a single machine.
Juber On Fri, Jul 30, 2010 at 10:27 AM, Gaurav Sharma <[email protected]> wrote: > Juber - I am wondering if you really do need MR/hadoop for this use case? As > you have also indicated, a standalone method might be able to do the job for > you. > > On Fri, Jul 30, 2010 at 1:45 AM, juber patel <[email protected]> wrote: >> >> thanks Ken for the reply. >> >> well, that's what I am doing right now. But the output from mappers >> needs to be processed together. Due to the nature of the problem, >> sorting is trivial once the map output becomes available. That's why I >> don't want to spend time in hadoop's inbuilt sort that involves disk >> access. (My sort will be in memory. The number of all possible sort >> keys is already known). >> >> Another problem is that this output needs to be processed further. So, >> there is unnecessary disk write and read. Again, my premise is that >> will be the slowest part of the program. So, I want a single reducer >> or a standalone program that will further process the map output, but >> I want to avoid disk i/o. >> >> thanks, >> >> Juber >> >> >> On Fri, Jul 30, 2010 at 12:36 AM, Ken Goodhope <[email protected]> >> wrote: >> > If you don't need sorted input, then you probably don't even need a >> > reducer. Try putting all your functionality in the mapper and then set >> > reduce tasks to zero. >> > >> > On Thursday, July 29, 2010, juber patel <[email protected]> wrote: >> >> Hi, >> >> >> >> Is it possible to use hadoop and not use disk i/o, apart from the >> >> initial input? >> >> >> >> I am asking this with the assumption that disk i/o is the bottleneck >> >> in overall processing, even more than the network access if you are on >> >> a dedicated, high speed cluster. (Does anyone have experience to >> >> confirm or reject this assumption?) >> >> >> >> I know that my programs logic does not require disk access after >> >> initial input. I don't even require sorting, but would like to combine >> >> the mapper output to reduce its size. This output is fed to another >> >> job/standalone program where it is interpreted meaningfully. I know >> >> this job/standalone program could be the reducer, but I don't want to >> >> spend time in sorting, especially involving disk spills. It is not >> >> required. >> >> >> >> Does anyone have a suggestion for this scenario? Is there something >> >> like NetworkInputFormat? Is there a way to start the reduce phase as >> >> mapper output starts coming in? I am thinking in terms of blocking >> >> queues, without disk access but with hadoop's fault tolerance, input >> >> splitting etc. >> >> >> >> >> >> thanks in advance, >> >> >> >> Juber >> >> >> > > >
