Eric J. Christeson wrote:
On Mar 19, 2009, at 12:03 PM, Sami Siren wrote:
Andrzej Bialecki wrote:
How about the following: we build just 2 packages:
* binary: this includes only base hadoop libs in lib/ (enough to
start a local job, no optional filesystems etc), the *.job and *.war
files and scripts. Scripts would check for the presence of plugins/
dir, and offer an option to create it from *.job. Assumption here is
that this shouldbe enough to run full cycle in local mode, and that
people who want to run a distributed cluster will first install a
plain Hadoop release, and then just put the *.job and bin/nutch on
the master.
* source: no build artifacts, no .svn (equivalent to svn export),
simple tgz.
this sounds good to me. additionally some new documentation needs to
be written too.
Distributed is a little more complicated than just dropping *.job and
bin/nutch on a hadoop install. Will this even work unless one edits
config/<stuff> and builds a new .job? Anyone using distributed nutch
probably wouldn't be interested in something trivial so a step-by-step
config how-to would probably be a good idea.
Actually, this works very well and it _is_ just a matter of dropping the
*.job file and a (slightly) modified bin/nutch.
Some time ago I committed a fix that removed Hadoop artifacts from nutch
*.job file. This was exactly to avoid confusion that multiple
hadoop-site.xml and hadoop*.jar caused (one in your Hadoop install and
the other in your Nutch job jar). So now the only place where you should
edit Hadoop-related stuff is in your Hadoop conf/ dir, and the only
place where you should edit Nutch-related stuff is in your Nutch conf/
dir (and after that indeed you need to rebuild the *.job jar and drop
the new version to your Hadoop master).
--
Best regards,
Andrzej Bialecki <><
___. ___ ___ ___ _ _ __________________________________
[__ || __|__/|__||\/| Information Retrieval, Semantic Web
___|||__|| \| || | Embedded Unix, System Integration
http://www.sigram.com Contact: info at sigram dot com
