This seems really confusing to me:
<snip from="RecommenderJob">
Option numReccomendationsOpt = AbstractJob.buildOption("numRecommendations",
"n",
"Number of recommendations per user", "10");
Option usersFileOpt = AbstractJob.buildOption("usersFile", "u",
"File of users to recommend for", null);
Option booleanDataOpt = AbstractJob.buildOption("booleanData", "b",
"Treat input as without pref values", Boolean.FALSE.toString());
Map<String,String> parsedArgs = AbstractJob.parseArguments(
args, numReccomendationsOpt, usersFileOpt, booleanDataOpt);
if (parsedArgs == null) {
return -1;
}
Configuration originalConf = getConf();
Path inputPath = new Path(originalConf.get("mapred.input.dir"));
Path outputPath = new Path(originalConf.get("mapred.output.dir"));
</snip>
So, in some parts, I need to have Hadoop options configured (presumably either
in a Conf file or via -D) while other inputs
I'm going to put in with the traditional -- stuff.
-Grant
On May 28, 2010, at 2:05 PM, Sean Owen wrote:
> I'm for all of those ideas. Would be great if someone else makes changes to
> make it more broadly usable since so far its just structure I have chucked
> in.
>
> On May 28, 2010 12:16 PM, "Drew Farris" <[email protected]> wrote:
>
> [email protected] [email protected]
>
> It might be nice to add a few default flags to AbstractJob that map directly
> to -D arguments in hadoop, for example, I could see having -i map to
> -Dmapred.input.dir, -o to -Dmapred.output.dir, -nr -Dmapred.num.reducers
> etc.. I think it is great to be able to accept arbitrary -D arguments but it
> would be nice to accept shorthand which also gets displayed in -h output.
>
> The -D options don't get included in -h and as a result it is unclear just
> how to specify input or output to someone who might not be too familliar
> with hadoop conventions.
>
> From the API perspective, AbstractJob could provide no-arg methods like
> AbstractJob.buildInputOption() etc, where the class using the AbstractJob
> api need not be concerned with the precise letters, parameters, description
> required for the option.
>
> Tangentially related, I was wondering something about AbstractJob: With the
> advent of the parsedArgs map returned by AbstractJob.parseArguments is there
> a need to pass Option arguments around anymore? Could AbstractJob maintain
> Options state in a sense?
>
> For example, from RecommenderJob:
>
> Option numReccomendationsOpt =
> AbstractJob.buildOption("numRecommendations", "n",
> "Number of recommendations per user", "10");
> Option usersFileOpt = AbstractJob.buildOption("usersFile", "u",
> "File of users to recommend for", null);
> Option booleanDataOpt = AbstractJob.buildOption("booleanData", "b",
> "Treat input as without pref values", Boolean.FALSE.toString());
>
> Map<String,String> parsedArgs = AbstractJob.parseArguments(
> args, numReccomendationsOpt, usersFileOpt, booleanDataOpt);
> if (parsedArgs == null) {
> return -1;
> }
>
> Could be changed to something like:
>
> buildOption("numRecommendations", "n", "Number of recommendations per user",
> "10");
> buildOption("usersFile", "u", "File of users to recommend for", null);
> buildOption("booleanData", "b", "Treat input as without pref values",
> Boolean.FALSE.toString());
> Map<String,String> parsedArgs = parseArguments();
>
> Providing a set of input validators that check the input before launching a
> job sounds like a pretty cool idea too.
>
>
> On Fri, May 28, 2010 at 10:55 AM, Sean Owen <[email protected]> wrote:
>
>> Does it help to note this ...
--------------------------
Grant Ingersoll
http://www.lucidimagination.com/
Search the Lucene ecosystem using Solr/Lucene:
http://www.lucidimagination.com/search
