Yes its as simple as that. The JobTracker takes care of delegation of tasks, therefore there is no need for Nutch to be present on every node. Hadoop and HBase (or whichever back you choose) is a different case.
On Sunday, June 30, 2013, Tejas Patil <[email protected]> wrote: > I have never used 2.x on prod but this is what I would do: > The datastore backend needs to be setup on the cluster. Even Hadoop must be > installed. Export all relevant environment variables. Nutch 2.x source must > be downloaded to the master node. Then modify the required configs and run > "ant runtime" to create nutch binaries inside "NUTCH_HOME/runtime/deploy". > Trigger the crawl command from the master node. > > > On Sun, Jun 30, 2013 at 1:16 AM, Tony Mullins <[email protected] >wrote: > >> Tejas , that [0] is for nutch 1.x which uses hdfs for its data storage. And >> as new nutch 2.x uses hbase (backend) which is already based on hadoop >> (hdfs). If I deploy my hbase in cluster mode on 3 different nodes then do I >> still need to deploy nutch 2.x on these 3 nodes as well ? >> >> Could you please care to add some little more information for nutch2.x + >> hbase + hadoop ? >> >> Regards, >> Khan >> >> [0] : http://wiki.apache.org/nutch/NutchHadoopTutorial >> >> >> On Sat, Jun 29, 2013 at 10:50 PM, Tejas Patil <[email protected] >> >wrote: >> >> > On Sat, Jun 29, 2013 at 10:36 AM, imran khan <[email protected] >> > >wrote: >> > >> > > Greetings, >> > > >> > > Is there any guide for setting up nutch2.x in cluster mode ? >> > > >> > >> > [0] is a relevant wiki page .. which has not been updated since a long >> > time. >> > I am guessing that you have already tried running in local mode as given >> in >> > [1]. For cluster mode, have hadoop 1.2.0 setup and its variables >> exported, >> > set nutch configs as per your requirements, run 'ant' and then run nutch >> > commands from $NUTCH_HOME/runtime/deploy >> > >> > >> > > And which versions of hadopp nutch2.x/hbase works well in cluster mode >> ? >> > > >> > >> > Use Nutch 2.2 and HBase 0.90.x >> > >> > > >> > > Regards, >> > > Khan >> > > >> > >> > [0] : http://wiki.apache.org/nutch/NutchHadoopTutorial >> > [1] : http://wiki.apache.org/nutch/Nutch2Tutorial >> > >> > -- *Lewis*
