Re: nutch-2.x with hbase filter option

[alxsss]

 

Hi,

It turned out that the error is generated because of the mismatch of the new 
function's return type(Integer) and the resulting variable. After changing 
these lines 
in 

src/java/org/apache/nutch/crawl/AdaptiveFetchSchedule.java
CHANGE 


 int interval = page.getFetchInterval();
    switch (state) {
      case FetchSchedule.STATUS_MODIFIED:
        interval *= (1.0f - DEC_RATE);
        break;
      case FetchSchedule.STATUS_NOTMODIFIED:
        interval *= (1.0f + INC_RATE);
        break;
      case FetchSchedule.STATUS_UNKNOWN:
        break;
    }

 

 TO
  Integer interval = page.getFetchInterval();
    switch (state) {
      case FetchSchedule.STATUS_MODIFIED:
        interval *=(int) (1.0f - DEC_RATE);
        break;
      case FetchSchedule.STATUS_NOTMODIFIED:
        interval *=(int) (1.0f + INC_RATE);
        break;
      case FetchSchedule.STATUS_UNKNOWN:
        break;
    }

---------------------------------------------------------------
src/java/org/apache/nutch/indexer/IndexingJob.java 
//complains  NoSuchMethodError  getMinorCode() 
CHANGE
  if (pstatus == null || !ParseStatusUtils.isSuccess(pstatus)
          || pstatus.getMinorCode() == ParseStatusCodes.SUCCESS_REDIRECT) {
        return; // filter urls not parsed
      }

TO
if (pstatus == null || !ParseStatusUtils.isSuccess(pstatus)

          || (int)pstatus.getMinorCode() == ParseStatusCodes.SUCCESS_REDIRECT) {
        return; // filter urls not parsed
      }
-----------------------------------------------------------------
src/java/org/apache/nutch/indexer/IndexUtil.java

//complains  NoSuchMethodError  getBatchId() 
CHANGE 
if (page.getBatchId() != null) {
      doc.add("batchId", page.getBatchId().toString());
    }

TO

 if ((Utf8)page.getBatchId() != null) {
      doc.add("batchId", page.getBatchId().toString());
    }

import org.apache.avro.util.Utf8;
-----------------------------------------------------------------------

With these additional changes it works so far :).


Thanks.
Alex.


-----Original Message-----
From: Alparslan Avcı <alparslan.a...@agmlab.com>
To: user <user@nutch.apache.org>
Sent: Thu, Apr 10, 2014 7:09 am
Subject: Re: nutch-2.x with hbase filter option


Hi Alex,

It seems from the log that your build's WebPage class has no 
getFetchInterval() method. However,that method exists in 2.x branch and 
the patch in NUTCH-1714.

Have you changed any code in WebPage class? And do you run in local or 
distributed mode?


Thanks,
Alparslan


On 10-04-2014 01:07, alxsss wrote:
> Hi,
>
> I was able to fix these errors making some changes to code and using
> avro-1.7.
> Now when I run updatedb command it gives
>
> 2014-04-09 14:29:36,460 FATAL org.apache.hadoop.mapred.Child: Error running
> child : java.lang.NoSuchMethodError:
> org.apache.nutch.storage.WebPage.getFetchInterval()I
>          at
> org.apache.nutch.crawl.AdaptiveFetchSchedule.setFetchSchedule(AdaptiveFetchSchedule.java:85)
>          at
> org.apache.nutch.crawl.DbUpdateReducer.reduce(DbUpdateReducer.java:141)
>          at
> org.apache.nutch.crawl.DbUpdateReducer.reduce(DbUpdateReducer.java:41)
>          at org.apache.hadoop.mapreduce.Reducer.run(Reducer.java:176)
>          at
> org.apache.hadoop.mapred.ReduceTask.runNewReducer(ReduceTask.java:649)
>          at org.apache.hadoop.mapred.ReduceTask.run(ReduceTask.java:417)
>          at org.apache.hadoop.mapred.Child$4.run(Child.java:255)
>          at java.security.AccessController.doPrivileged(Native Method)
>          at javax.security.auth.Subject.doAs(Subject.java:396)
>          at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1121)
>          at org.apache.hadoop.mapred.Child.main(Child.java:249)
>   
>
> Thanks.
> Alex.
>
>
>
> --
> View this message in context: 
> http://lucene.472066.n3.nabble.com/nutch-2-x-with-hbase-filter-option-tp4121242p4130227.html
> Sent from the Nutch - User mailing list archive at Nabble.com.