I'm for all of those ideas. Would be great if someone else makes changes to make it more broadly usable since so far its just structure I have chucked in.
On May 28, 2010 12:16 PM, "Drew Farris" <[email protected]> wrote: [email protected] [email protected] It might be nice to add a few default flags to AbstractJob that map directly to -D arguments in hadoop, for example, I could see having -i map to -Dmapred.input.dir, -o to -Dmapred.output.dir, -nr -Dmapred.num.reducers etc.. I think it is great to be able to accept arbitrary -D arguments but it would be nice to accept shorthand which also gets displayed in -h output. The -D options don't get included in -h and as a result it is unclear just how to specify input or output to someone who might not be too familliar with hadoop conventions. >From the API perspective, AbstractJob could provide no-arg methods like AbstractJob.buildInputOption() etc, where the class using the AbstractJob api need not be concerned with the precise letters, parameters, description required for the option. Tangentially related, I was wondering something about AbstractJob: With the advent of the parsedArgs map returned by AbstractJob.parseArguments is there a need to pass Option arguments around anymore? Could AbstractJob maintain Options state in a sense? For example, from RecommenderJob: Option numReccomendationsOpt = AbstractJob.buildOption("numRecommendations", "n", "Number of recommendations per user", "10"); Option usersFileOpt = AbstractJob.buildOption("usersFile", "u", "File of users to recommend for", null); Option booleanDataOpt = AbstractJob.buildOption("booleanData", "b", "Treat input as without pref values", Boolean.FALSE.toString()); Map<String,String> parsedArgs = AbstractJob.parseArguments( args, numReccomendationsOpt, usersFileOpt, booleanDataOpt); if (parsedArgs == null) { return -1; } Could be changed to something like: buildOption("numRecommendations", "n", "Number of recommendations per user", "10"); buildOption("usersFile", "u", "File of users to recommend for", null); buildOption("booleanData", "b", "Treat input as without pref values", Boolean.FALSE.toString()); Map<String,String> parsedArgs = parseArguments(); Providing a set of input validators that check the input before launching a job sounds like a pretty cool idea too. On Fri, May 28, 2010 at 10:55 AM, Sean Owen <[email protected]> wrote: > Does it help to note this ...
