Good call. Thank you guys for helping me out. I'll do some experiments on efficiency later and keep you guys updated.
- Chang On Mon, Mar 24, 2008 at 6:51 PM, Riccardo Boscolo <[EMAIL PROTECTED]> wrote: > That's simple, add a combiner that looks exactly like your reducer, but > collects IntWritable(s) as values. Btw, yours is a very good example, > because it is often forgotten that the (key,value) class types of > combiners > should always match those of mappers. > > RB > > On Mon, Mar 24, 2008 at 3:42 PM, Chang Hu <[EMAIL PROTECTED]> wrote: > > > Thanks Doug! I am able to run the job after removing the setConbiner() > > line. Does it hurt efficiency and how do I add a combiner? > > > > - Chang > > > > On Mon, Mar 24, 2008 at 6:26 PM, Doug Cutting <[EMAIL PROTECTED]> > wrote: > > > > > Chang Hu wrote: > > > > Code below, also attached. I put this together from the word count > > > > example. > > > > > > The problem is with your combiner. When a combiner is specified, it > > > generates the final map output, since combination is a map-side > > > operation. Your combiner takes <Text,IntWritable> generated by your > > > mapper and generates <Text,Text> outputs, which would not be > acceptable > > > to your reducer, which accepts <Text,IntWritable> inputs. Does that > > > make sense? > > > > > > Doug > > > > > > > > > > > -- > > --------------- > > Überstehen ist alles. > > > > > > Chang Hu > > Ph.D. student > > Computer Science Department > > University of Maryland > > > > > > -- > ------------------------------- > Riccardo Boscolo, PhD > V.P. of Core Technology > Netseer Inc. > 11943 Montana Ave, Suite 200 > Los Angeles, CA 90049 > T: 310-597-4482 > F: 310-597-4489 > Email: [EMAIL PROTECTED] > ------------------------------- > -- --------------- Überstehen ist alles. Chang Hu Ph.D. student Computer Science Department University of Maryland
