Hi Matt, I don't know if you got my message on the github mirrior issue. If you could get the patch uploaded to a new Nutch Jira ticket (unless one is already open) then I will be very happy to test as some free time now means I am able to test a few patches.
Thanks Lewis On Sat, Sep 8, 2012 at 8:36 PM, Matt MacDonald <[email protected]> wrote: > Hi, > > Updated my local with the following > https://github.com/apache/nutch/pull/1/files and the crawler is > adhering to the db.ignore.external.links property when encountering > local redirects that point to external domains. > > Thanks, > Matt > > On Sat, Sep 8, 2012 at 1:44 PM, Matt MacDonald <[email protected]> wrote: >> Hi, >> >> Likely this is a question for Ferdy but if anyone else has input >> that'd be great. When running a crawl that I would expect to be >> contained to a single domain I'm seeing the crawler jump out to other >> domains. I'm using the trunk of Nutch 2.x which includes the following >> commit: >> https://github.com/apache/nutch/commit/c5e2236f36a881ee7fec97aff3baf9bb32b40200 >> >> The goal is to perform a focused crawl against a single domain and >> restrict the crawler from expanding beyond that domain. I've set the >> db.ignore.external.links property to true. I do not want to add a >> regex to regex-urlfilter.txt as I will be adding several thousand >> urls. The domain that I am crawling has documents with outlinks that >> are still within the domain but then redirect to external domains. >> >> cat urls/seed.txt >> http://www.ci.watertown.ma.us/ >> >> cat conf/nutch-site.xml >> ... >> <property> >> <name>db.ignore.external.links</name> >> <value>true</value> >> <description>If true, outlinks leading from a page to external hosts >> will be ignored. This is an effective way to limit the crawl to include >> only initially injected hosts, without creating complex URLFilters. >> </description> >> </property> >> >> <property> >> <name>plugin.includes</name> >> >> <value>protocol-http|urlfilter-regex|parse-(html|tika)|index-(basic|anchor)|urlnormalizer-(pass|regex|basic)|scoring-opic</value> >> <description>Regular expression naming plugin directory names to >> include. Any plugin not matching this expression is excluded. >> In any case you need at least include the nutch-extensionpoints plugin. >> By >> default Nutch includes crawling just HTML and plain text via HTTP, >> and basic indexing and search plugins. In order to use HTTPS please >> enable >> protocol-httpclient, but be aware of possible intermittent >> problems with the >> underlying commons-httpclient library. >> </description> >> </property> >> ... >> >> Running >> bin/nutch crawl urls -depth 8 -topN 100000 >> >> results in the the crawl eventually fetching and parsing documents on >> domains external to the only link in the seed.txt file. >> >> I would not expect to see urls like the following in my logs and in >> the HBase webpage table: >> >> fetching http://www.masshome.com/tourism.html >> Parsing http://www.disabilityinfo.org/ >> >> I'm reviewing the code changes but am still getting up to speed on the >> code base. Any ideas while I continue to dig around? Configuration >> issue or code? >> >> Thanks, >> Matt -- Lewis
