Juber - I am wondering if you really do need MR/hadoop for this use case? As you have also indicated, a standalone method might be able to do the job for you.
On Fri, Jul 30, 2010 at 1:45 AM, juber patel <[email protected]> wrote: > thanks Ken for the reply. > > well, that's what I am doing right now. But the output from mappers > needs to be processed together. Due to the nature of the problem, > sorting is trivial once the map output becomes available. That's why I > don't want to spend time in hadoop's inbuilt sort that involves disk > access. (My sort will be in memory. The number of all possible sort > keys is already known). > > Another problem is that this output needs to be processed further. So, > there is unnecessary disk write and read. Again, my premise is that > will be the slowest part of the program. So, I want a single reducer > or a standalone program that will further process the map output, but > I want to avoid disk i/o. > > thanks, > > Juber > > > On Fri, Jul 30, 2010 at 12:36 AM, Ken Goodhope <[email protected]> > wrote: > > If you don't need sorted input, then you probably don't even need a > > reducer. Try putting all your functionality in the mapper and then set > > reduce tasks to zero. > > > > On Thursday, July 29, 2010, juber patel <[email protected]> wrote: > >> Hi, > >> > >> Is it possible to use hadoop and not use disk i/o, apart from the > >> initial input? > >> > >> I am asking this with the assumption that disk i/o is the bottleneck > >> in overall processing, even more than the network access if you are on > >> a dedicated, high speed cluster. (Does anyone have experience to > >> confirm or reject this assumption?) > >> > >> I know that my programs logic does not require disk access after > >> initial input. I don't even require sorting, but would like to combine > >> the mapper output to reduce its size. This output is fed to another > >> job/standalone program where it is interpreted meaningfully. I know > >> this job/standalone program could be the reducer, but I don't want to > >> spend time in sorting, especially involving disk spills. It is not > >> required. > >> > >> Does anyone have a suggestion for this scenario? Is there something > >> like NetworkInputFormat? Is there a way to start the reduce phase as > >> mapper output starts coming in? I am thinking in terms of blocking > >> queues, without disk access but with hadoop's fault tolerance, input > >> splitting etc. > >> > >> > >> thanks in advance, > >> > >> Juber > >> > > >
