Owen, It works, thank you for your help.
Jeff Zhang On Tue, Nov 24, 2009 at 8:36 AM, Jeff Zhang <[email protected]> wrote: > > You're right, I will try that. > > Thank you > > > Jeff Zhang > > > > On Mon, Nov 23, 2009 at 9:19 AM, Owen O'Malley <[email protected]> wrote: > >> >> On Nov 22, 2009, at 4:48 PM, Jeff Zhang wrote: >> >> My concern is that it is just like hard code to use >>> conf.setNumReduceTasks >>> on the configuration. It is not flexible, so my idea is that adding an >>> interface to change the reducer number dynamically according the >>> different >>> size of input data set. >>> >> >> You misunderstand. I meant doing something like: >> >> public class MyInputFormat .... >> >> public InputSplit[] getSplits(JobConf conf) { >> InputSplit[] result = ...; >> // compute total size of input >> conf.setNumReduceTasks(max(6, size / 10G)); >> } >> } >> >> I haven't checked the code to make sure it will work, but I believe it >> will. >> >> -- Owen >> > >
