thanks Ken for the reply. well, that's what I am doing right now. But the output from mappers needs to be processed together. Due to the nature of the problem, sorting is trivial once the map output becomes available. That's why I don't want to spend time in hadoop's inbuilt sort that involves disk access. (My sort will be in memory. The number of all possible sort keys is already known).
Another problem is that this output needs to be processed further. So, there is unnecessary disk write and read. Again, my premise is that will be the slowest part of the program. So, I want a single reducer or a standalone program that will further process the map output, but I want to avoid disk i/o. thanks, Juber On Fri, Jul 30, 2010 at 12:36 AM, Ken Goodhope <[email protected]> wrote: > If you don't need sorted input, then you probably don't even need a > reducer. Try putting all your functionality in the mapper and then set > reduce tasks to zero. > > On Thursday, July 29, 2010, juber patel <[email protected]> wrote: >> Hi, >> >> Is it possible to use hadoop and not use disk i/o, apart from the >> initial input? >> >> I am asking this with the assumption that disk i/o is the bottleneck >> in overall processing, even more than the network access if you are on >> a dedicated, high speed cluster. (Does anyone have experience to >> confirm or reject this assumption?) >> >> I know that my programs logic does not require disk access after >> initial input. I don't even require sorting, but would like to combine >> the mapper output to reduce its size. This output is fed to another >> job/standalone program where it is interpreted meaningfully. I know >> this job/standalone program could be the reducer, but I don't want to >> spend time in sorting, especially involving disk spills. It is not >> required. >> >> Does anyone have a suggestion for this scenario? Is there something >> like NetworkInputFormat? Is there a way to start the reduce phase as >> mapper output starts coming in? I am thinking in terms of blocking >> queues, without disk access but with hadoop's fault tolerance, input >> splitting etc. >> >> >> thanks in advance, >> >> Juber >> >
