Re: nutch-2.x with hbase filter option

alxsss Mon, 31 Mar 2014 13:00:13 -0700

I  have applied the patch to the current trunk

Here is the output of ant


    [javac] Compiling 2 source files to 
/home/kingson/nutch-2.x-svn/build/classes
    [javac] 
/nutch-2.x/src/java/org/apache/nutch/host/HostDbUpdateReducer.java:78: cannot 
find symbol
    [javac] symbol  : method newBuilder()
    [javac] location: class org.apache.nutch.storage.Host
    [javac]     Host host = Host.newBuilder().build();
    [javac]                     ^
    [javac] /nutch-2.x/src/java/org/apache/nutch/host/HostInjectorJob.java:124: 
cannot find symbol
    [javac] symbol  : method newBuilder()
    [javac] location: class org.apache.nutch.storage.Host
    [javac]       Host host = Host.newBuilder().build();
    [javac]                       ^
    [javac] 2 errors

BUILD FAILED
/nutch-2.x/build.xml:101: Compile failed; see the compiler error output for 
details.


Also, here are files that import/use StateManager class, which seems was 
removed from GORA_94

$ grep -r StateManager src/
src/java/org/apache/nutch/storage/Host.java:import 
org.apache.gora.persistency.StateManager;
src/java/org/apache/nutch/storage/Host.java:import 
org.apache.gora.persistency.impl.StateManagerImpl;
src/java/org/apache/nutch/storage/Host.java:    this(new StateManagerImpl());
src/java/org/apache/nutch/storage/Host.java:  public Host(StateManager 
stateManager) {
src/java/org/apache/nutch/storage/Host.java:  public Host 
newInstance(StateManager stateManager) {
src/java/org/apache/nutch/storage/Host.java:    
getStateManager().setDirty(this, _field);
src/java/org/apache/nutch/storage/Host.java:    
getStateManager().setDirty(this, 0);
src/java/org/apache/nutch/storage/Host.java:    
getStateManager().setDirty(this, 0);
src/java/org/apache/nutch/storage/Host.java:    
getStateManager().setDirty(this, 1);
src/java/org/apache/nutch/storage/Host.java:    
getStateManager().setDirty(this, 1);
src/java/org/apache/nutch/storage/Host.java:    
getStateManager().setDirty(this, 2);
src/java/org/apache/nutch/storage/Host.java:    
getStateManager().setDirty(this, 2);
src/java/org/apache/nutch/storage/WebPage.java:import 
org.apache.gora.persistency.StateManager;
src/java/org/apache/nutch/storage/WebPage.java:import 
org.apache.gora.persistency.impl.StateManagerImpl;
src/java/org/apache/nutch/storage/WebPage.java:    this(new StateManagerImpl());
src/java/org/apache/nutch/storage/WebPage.java:  public WebPage(StateManager 
stateManager) {
src/java/org/apache/nutch/storage/WebPage.java:  public WebPage 
newInstance(StateManager stateManager) {
src/java/org/apache/nutch/storage/WebPage.java:    
getStateManager().setDirty(this, _field);
src/java/org/apache/nutch/storage/WebPage.java:    
getStateManager().setDirty(this, 18);
src/java/org/apache/nutch/storage/WebPage.java:    
getStateManager().setDirty(this, 18);
src/java/org/apache/nutch/storage/WebPage.java:    
getStateManager().setDirty(this, 19);
src/java/org/apache/nutch/storage/WebPage.java:    
getStateManager().setDirty(this, 19);
src/java/org/apache/nutch/storage/WebPage.java:    
getStateManager().setDirty(this, 20);
src/java/org/apache/nutch/storage/WebPage.java:    
getStateManager().setDirty(this, 20);
src/java/org/apache/nutch/storage/WebPage.java:    
getStateManager().setDirty(this, 21);
src/java/org/apache/nutch/storage/WebPage.java:    
getStateManager().setDirty(this, 21);
src/java/org/apache/nutch/storage/WebPage.java:    
getStateManager().setDirty(this, 22);
src/java/org/apache/nutch/storage/WebPage.java:    
getStateManager().setDirty(this, 22);
src/java/org/apache/nutch/storage/ParseStatus.java:import 
org.apache.gora.persistency.StateManager;
src/java/org/apache/nutch/storage/ParseStatus.java:import 
org.apache.gora.persistency.impl.StateManagerImpl;
src/java/org/apache/nutch/storage/ParseStatus.java:    this(new 
StateManagerImpl());
src/java/org/apache/nutch/storage/ParseStatus.java:  public 
ParseStatus(StateManager stateManager) {
src/java/org/apache/nutch/storage/ParseStatus.java:  public ParseStatus 
newInstance(StateManager stateManager) {
src/java/org/apache/nutch/storage/ParseStatus.java:    
getStateManager().setDirty(this, _field);
src/java/org/apache/nutch/storage/ParseStatus.java:    
getStateManager().setDirty(this, 2);
src/java/org/apache/nutch/storage/ProtocolStatus.java:import 
org.apache.gora.persistency.StateManager;
src/java/org/apache/nutch/storage/ProtocolStatus.java:import 
org.apache.gora.persistency.impl.StateManagerImpl;
src/java/org/apache/nutch/storage/ProtocolStatus.java:    this(new 
StateManagerImpl());
src/java/org/apache/nutch/storage/ProtocolStatus.java:  public 
ProtocolStatus(StateManager stateManager) {
src/java/org/apache/nutch/storage/ProtocolStatus.java:  public ProtocolStatus 
newInstance(StateManager stateManager) {
src/java/org/apache/nutch/storage/ProtocolStatus.java:    
getStateManager().setDirty(this, _field);
src/java/org/apache/nutch/storage/ProtocolStatus.java:    
getStateManager().setDirty(this, 1);

Thanks.
Alex.

 

 

 

-----Original Message-----
From: Lewis John Mcgibbney <[email protected]>
To: user <[email protected]>
Sent: Sun, Mar 30, 2014 3:37 am
Subject: Re: nutch-2.x with hbase filter option


Hi alxsss,

On Sat, Mar 29, 2014 at 10:15 PM, <[email protected]> wrote:

>
> I downloaded GORA_94 branch and with libs from it a get
>>
>> 14/03/27 11:21:19 INFO crawl.InjectorJob: InjectorJob: Injecting urlDir:
>> test_urls
>> Exception in thread "main" java.lang.NoClassDefFoundError:
>> org/apache/gora/persistency/StateManager
>>
>
Please see the patch in https://issues.apache.org/jira/browse/NUTCH-1714
There has been a SIGNIFICANT amount of work done to the persistency API in
GORA_94 this has the consequence of meaning that API usage in Nutch has
also changed a bit.
Let us know how you get on.
Also thanks Alparslan for the patch in NUTCH-1714
Thanks
Lewis

Re: nutch-2.x with hbase filter option

Reply via email to