Hi Phil,

> -----Original Message-----
> From: Phil Barnett [mailto:ph...@philb.us]
> Sent: Wednesday, 21 April 2010 8:39 AM
> To: nutch-user@lucene.apache.org
> Subject: Question about crawler.
> Is there some place to tell why the crawler has rejected a page? I'm
> trying
> to get 1.1 working and basically it doesn't seem to crawl the same way
> that
> 1.0 does.
> I have tika included in the parse- section of conf/nutch-site.xml
> I have DEBUG set for all the crawl sections, but it doesn't really say
> why
> it's rejecting a site.
> I have the crawler set to not follow external links and I seed the top
> level
> of each site.
> I'm just unclear on how to proceed to troubleshoot this.

Nutch observes robots.txt instructions. Examine robots.txt files of sites that 
get rejected. The cause may be there. You mention that 1.0 used to crawl these 
sites OK. Did you copy its configuration to 1.1? In particular, the agent id 

If this looks OK and DEBUG output does not tell you enough, you can insert 
extra output in URL filters involved.



Reply via email to