In all honesty I would make sure that you have a local and up-to-date nutch-$version.job file generated and try it out in runtime/local before using the job in /runtime/deploy on your cluster. You will know if it is good to go or not. When you are ready to deploy it to your cluster (e.g. once your satisfied that it works on a test/sub set crawl) setup then just make it available to your Hadoop Job tracker classpath.
On Thu, May 30, 2013 at 3:48 PM, Yves S. Garret <[email protected]>wrote: > I have $HADOOP_INSTALL in my path, would this be enough Lewis? Or > would I need to copy around some jar files? > > > On Thu, May 30, 2013 at 6:35 PM, Lewis John Mcgibbney < > [email protected]> wrote: > > > Make sure that everything is compiled and you are running from runtime or > > with the Jar in hadoop > > > > > > On Thu, May 30, 2013 at 3:00 PM, Yves S. Garret > > <[email protected]>wrote: > > > > > Here is my hbase-site.xml: > > > http://bin.cakephp.org/view/2054577438 > > > > > > I've set this property as well. > > > > > > > > > On Thu, May 30, 2013 at 5:57 PM, Shah, Nishant <[email protected]> > > wrote: > > > > > > > What about your storage.data.store.class property in nutch-site.xml > ? I > > > > think you have to change the value to use hbase. For me it is > > > > org.apache.gora.hbase.store.HBasetore. > > > > > > > > -----Original Message----- > > > > From: Yves S. Garret [mailto:[email protected]] > > > > Sent: Thursday, May 30, 2013 2:52 PM > > > > To: [email protected] > > > > Subject: Re: How to setup HBase as backend > > > > > > > > Yes. For the moment, for simplicity sake, I have everything going to > > > /tmp. > > > > > > > > hbase(main):004:0> scan 'test' > > > > ROW > > > > COLUMN+CELL > > > > > > > > 0 row(s) in 0.2370 seconds > > > > > > > > I _should_ have a table "webpage being created when I run Nutch. > > > > > > > > > > > > On Thu, May 30, 2013 at 5:23 PM, Shah, Nishant <[email protected]> > > > wrote: > > > > > > > > > Is your hbase running ? > > > > > > > > > > -----Original Message----- > > > > > From: Yves S. Garret [mailto:[email protected]] > > > > > Sent: Thursday, May 30, 2013 2:18 PM > > > > > To: [email protected] > > > > > Subject: Re: How to setup HBase as backend > > > > > > > > > > Even when I do bin/nutch generate, this is what I get: > > > > > http://bin.cakephp.org/view/1815127825 > > > > > > > > > > > > > > > On Thu, May 30, 2013 at 5:14 PM, Yves S. Garret > > > > > <[email protected]>wrote: > > > > > > > > > > > Ok, similar issue: > > > > > > http://bin.cakephp.org/view/180499048 > > > > > > > > > > > > I've left the defaults for config as they were, except this is in > > > > > > gora.properties in apache nutch. > > > > > > gora.datastore.default=org.apache.gora.hbase.store.HBaseStore > > > > > > > > > > > > > > > > > > On Wed, May 29, 2013 at 7:40 PM, Lewis John Mcgibbney < > > > > > > [email protected]> wrote: > > > > > > > > > > > >> Yes, as Tejas mentioned, He runs fine with 0.90.6 API changes > make > > > > > >> more recent HBase versions incompatible. > > > > > >> We will be upgrading HBase API usage in Gora within the current > > > > > >> development drive. > > > > > >> Lewis > > > > > >> > > > > > >> > > > > > >> On Wed, May 29, 2013 at 4:36 PM, Yves S. Garret > > > > > >> <[email protected]>wrote: > > > > > >> > > > > > >> > Would HBase 0.90.X and Nutch 2.1 work? > > > > > >> > > > > > > >> > > > > > > >> > On Wed, May 29, 2013 at 5:05 PM, Lewis John Mcgibbney < > > > > > >> > [email protected]> wrote: > > > > > >> > > > > > > >> > > This is incompatible. > > > > > >> > > > > > > > >> > > > > > > > >> > > On Wed, May 29, 2013 at 1:59 PM, Yves S. Garret > > > > > >> > > <[email protected]>wrote: > > > > > >> > > > > > > > >> > > > Hi all, I'm using HBase 0.94.7 and Nutch 2.1. > > > > > >> > > > > > > > > >> > > > > > > > > >> > > > On Wed, May 29, 2013 at 4:55 PM, Adriana Farina > > > > > >> > > > <[email protected]>wrote: > > > > > >> > > > > > > > > >> > > > > Hi Yves, > > > > > >> > > > > > > > > > >> > > > > as Tejas said, your issue is almost certainly due to a > > > > > >> compatibility > > > > > >> > > > > problem between the version of Nutch and the one of > HBase. > > > > > >> > > > > > > > > > >> > > > > I had the same problem and in my case it was due to the > > > > > >> > > > > HBase > > > > > >> > version. > > > > > >> > > > > > > > > > >> > > > > I use Nutch 2.1 with HBase 0.90.4 and it works fine. > > > > > >> > > > > > > > > > >> > > > > > > > > > >> > > > > > > > > > >> > > > > 2013/5/29 Yves S. Garret <[email protected]> > > > > > >> > > > > > > > > > >> > > > > > Hi, I'm trying to run Nutch this time around with > HBase > > > > > >> > > > > > in the > > > > > >> > > > > background, > > > > > >> > > > > > as > > > > > >> > > > > > opposed to having MySQL instead. > > > > > >> > > > > > > > > > > >> > > > > > In the past, I followed this tutorial: > > > > > >> > > > > > http://nlp.solutions.asia/?p=180 > > > > > >> > > > > > > > > > > >> > > > > > This was all in good, but now that I have my HBase, > I'd > > > > > >> > > > > > like to > > > > > >> use > > > > > >> > > > that. > > > > > >> > > > > > I left the configuration of Nutch as it was and > > proceeded > > > > > >> > > > > > to > > > > > >> crawl > > > > > >> > > > > > nutch.apache.org. I got this error: > > > > > >> > > > > > http://bin.cakephp.org/view/1301117746 > > > > > >> > > > > > > > > > > >> > > > > > What am I doing wrong? > > > > > >> > > > > > > > > > > >> > > > > > At the moment, I'm reading through this, trying to get > > my > > > > > >> > > > > > stack > > > > > >> to > > > > > >> > > > work, > > > > > >> > > > > > will write back if I make any progress: > > > > > >> > > > > > > > > > > >> > > > > > > > > > >> > > > > > > > > >> > > > > > > > >> > > > > > > >> > > http://sujitpal.blogspot.com/2011/01/exploring-nutch-20-hbase-stora > > > > > >> ge > > > > > >> .html > > > > > >> > > > > > > > > > > >> > > > > > > > > > >> > > > > > > > > > >> > > > > > > > > > >> > > > > -- > > > > > >> > > > > Adriana Farina > > > > > >> > > > > > > > > > >> > > > > > > > > >> > > > > > > > >> > > > > > > > >> > > > > > > > >> > > -- > > > > > >> > > *Lewis* > > > > > >> > > > > > > > >> > > > > > > >> > > > > > >> > > > > > >> > > > > > >> -- > > > > > >> *Lewis* > > > > > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > -- > > *Lewis* > > > -- *Lewis*
