Thanks for the recommendation, haven't really looked into how the combiner might be able to help. Now, are there any downsides to having one 50GB file as an output? If I understand correctly, the number of reducers you set for your job is the number of files you will get as output.
2009/2/13 Amandeep Khurana <ama...@gmail.com> > What you can probably do is have the combine function do some reducing > before the single reducer starts off. That might help. > > > Amandeep Khurana > Computer Science Graduate Student > University of California, Santa Cruz > > > 2009/2/13 Kris Jirapinyo <kris.jirapi...@biz360.com> > > > I can't afford to have only one reducer as my dataset is huge...right now > > it > > is 50GB and so the output.collect() in the reducer will surely run out of > > java heap space. > > > > 2009/2/13 Amandeep Khurana <ama...@gmail.com> > > > > > Have only one instance of the reduce task. This will run once your map > > > tasks > > > are completed. You can set this in your job conf by using > > > conf.setNumReducers(1) > > > > > > > > > Amandeep Khurana > > > Computer Science Graduate Student > > > University of California, Santa Cruz > > > > > > > > > 2009/2/13 Kris Jirapinyo <kris.jirapi...@biz360.com> > > > > > > > What do you mean when I have only 1 reducer? > > > > > > > > On Fri, Feb 13, 2009 at 4:11 PM, Rasit OZDAS <rasitoz...@gmail.com> > > > wrote: > > > > > > > > > Kris, > > > > > This is the case when you have only 1 reducer. > > > > > If it doesn't have any side effects for you.. > > > > > > > > > > Rasit > > > > > > > > > > > > > > > 2009/2/14 Kris Jirapinyo <kjirapi...@biz360.com>: > > > > > > Is there a way to tell Hadoop to not run Map and Reduce > > concurrently? > > > > > I'm > > > > > > running into a problem where I set the jvm to Xmx768 and it seems > > > like > > > > 2 > > > > > > mappers and 2 reducers are running on each machine that only has > > > 1.7GB > > > > of > > > > > > ram, so it complains of not being able to allocate > memory...(which > > > > makes > > > > > > sense since 4x768mb > 1.7GB). So, if it would just finish the > Map > > > and > > > > > then > > > > > > start on Reduce, then there would be 2 jvm's running on one > machine > > > at > > > > > any > > > > > > given time and thus possibly avoid this out of memory error. > > > > > > > > > > > > > > > > > > > > > > > > > > -- > > > > > M. Raşit ÖZDAŞ > > > > > > > > > > > > > > >
