[ http://issues.apache.org/jira/browse/NUTCH-209?page=comments#action_12365800 ]
Andrzej Bialecki commented on NUTCH-209: ----------------------------------------- No problem. Re: plugin loading: well, when we are done building the binary distribution we already know for sure what plugins we have in the whole installation, so we might as well put this knowledge into a resource file (created by ant) that lists all available plugins in the installation, and then read this file instead of listing a directory. Today plugins already use a custom ClassLoader, so from this POV it would be a small change. Re: including only enabled plugins: potentially you would have to build a custom jar for each job, because the list of active plugins depends on the job's Configuration. I think I would prefer the replication trick. > include nutch jar in mapred jobs > -------------------------------- > > Key: NUTCH-209 > URL: http://issues.apache.org/jira/browse/NUTCH-209 > Project: Nutch > Type: Improvement > Versions: 0.8-dev > Reporter: Doug Cutting > Priority: Minor > Fix For: 0.8-dev > > I just added a simple way in Hadoop to specify the job jar file. When > constructing a JobConf one can specify a class whose containing jar is set to > be the job's jar. To take advantage of this in Nutch, we could add a util > class: > public class NutchJob extends JobConf { > public NutchJob(Configuration conf) { > super(conf, NutchJob.class); > } > } > Then change all of the places where we construct a JobConf to instead > construct a NutchJob. > Finally, we should add an ant target called 'job' that constructs a job jar, > containing all of the classes and the plugins, and make this the default > target. This way all Nutch code can be distributed with each job as it is > submitted, and daemons would only need to be restarted when Hadoop code is > updated. > Does this sound reasonable? -- This message is automatically generated by JIRA. - If you think it was sent incorrectly contact one of the administrators: http://issues.apache.org/jira/secure/Administrators.jspa - For more information on JIRA, see: http://www.atlassian.com/software/jira
