Thanks Markus, I'm very new to nutch. If you can point me in a direction on how to do this it would be nice. In the mean time I'll start reading up on how I might accomplish this.
See-ya Mitch On Thu, 2016-03-10 at 10:27 +0000, Markus Jelsma wrote: > Hi - if you have db.ignore.external.links set to true, and only inject > http://127.0.0.1:8080/cocoon into an empty CrawlDB, it should not try other > domains. So i am not sure what is really wrong, it should just work. I assume > you already have cpsc in your CrawlDB and don't filter it out. You should use > -filter (once) on the CrawlDB to remove cpsc. > > Markus > > > > -----Original message----- > > From:Mitch Baker <[email protected]> > > Sent: Wednesday 9th March 2016 22:40 > > To: [email protected] > > Subject: Only fetch 127.0.0.1:8080/* > > > > I have small setup to index some files on a local box. > > > > Solr 5 > > Nutch 1.11 > > > > I thought I had it configured to not try any URLs that are not local to > > the system but it still seems to look for them. > > > > fetching > > http://www.cpsc.gov/Media/Documents/Regulations-Laws--Standards/Advisory-Opinions/Wheelchairs-145--/ > > (queue crawl delay=2000ms) > > fetching http://www.cpsc.gov/PageFiles/121846/fuclearance.pdf (queue crawl > > delay=2000ms) > > fetching > > http://www.cpsc.gov/Business--Manufacturing/Business-Education/Business-Guidance/Phthalates-Information/ > > (queue crawl delay=2000ms) > > -activeThreads=150, spinWaiting=148, fetchQueues.totalSize=2091, > > fetchQueues.getQueueCount=1 > > fetching http://www.cpsc.gov/es/Research--Statistics/ (queue crawl > > delay=2000ms) > > > > The regex-urlfilter.txt: > > > > # Each non-comment, non-blank line contains a regular expression > > # prefixed by '+' or '-'. The first matching pattern in the file > > # determines whether a URL is included or ignored. If no pattern > > # matches, the URL is ignored. > > > > # skip file: ftp: and mailto: urls > > -^(file|ftp|mailto): > > > > # skip image and other suffixes we can't yet parse > > # for a more extensive coverage use the urlfilter-suffix plugin > > -\.(gif|GIF|jpg|JPG|png|PNG|ico|ICO|css|CSS|sit|SIT|eps|EPS|wmf|WMF|zip|ZIP|ppt|PPT|mpg|MPG|xls|XLS|gz|GZ|rpm|RPM|tgz|TGZ|mov|MOV|exe|EXE|jpeg|JPEG|bmp|BMP|js|JS)$ > > > > # skip URLs containing certain characters as probable queries, etc. > > #-[?*!@=] > > > > # skip URLs with slash-delimited segment that repeats 3+ times, to break > > loops > > -.*(/[^/]+)/[^/]+\1/[^/]+\1/ > > > > # skip specific PDF files in the volumes directory > > -.*00(FRONT|INTRO)\.PDF.* > > > > #skip > > #-^(http|https)://www\.*$ > > #-^(http|https)://blogs\.*$ > > #-^(http|https)://store\.*$ > > #-^(http|https)://.*\.google.com/.*$ > > #-^(http|https)://nist.gov/.*$ > > > > > > # accept anything else > > #+. > > +^http://127.0.0.1:8080/cocoon > > > > I have searched and tried several things, including nutch-site.xml: > > > > <configuration> > > <property> > > <name>http.agent.name</name> > > <value>nutch-solr-integration</value> > > </property> > > <property> > > <name>generate.max.per.host</name> > > <value>1000</value> > > </property> > > <property> > > <name>plugin.includes</name> > > <value>protocol-http|urlfilter-regex|parse-(html|tika|metatags)|index-(basic|anchor|metadata)|indexer-solr|query-(basic|site|url)|response-(json|xml)|summary-basic|scoring-opic|urlnormalizer-(pass|regex|basic)</value> > > </property> > > <property> > > <name>db.ignore.external.links</name> > > <value>true</value> > > <description>If true, outlinks leading from a page to external hosts or > > domain > > will be ignored. This is an effective way to limit the crawl to include > > only initially injected hosts, without creating complex URLFilters. > > See 'db.ignore.external.links.mode'. > > </description> > > </property> > > <property> > > <name>db.max.outlinks.per.page</name> > > <value>0</value> > > <description>The maximum number of outlinks that we'll process for a page. > > If this value is nonnegative (>=0), at most db.max.outlinks.per.page > > outlinks > > will be processed for a page; otherwise, all outlinks will be processed. > > </description> > > </property> > > <property> > > <name>fetcher.max.crawl.delay</name> > > <value>3</value> > > <description> > > If the Crawl-Delay in robots.txt is set to greater than this value (in > > seconds) then the fetcher will skip this page, generating an error report. > > If set to -1 the fetcher will never skip such pages and will wait the > > amount of time retrieved from robots.txt Crawl-Delay, however long that > > might be. > > </description> > > </property> > > <property> > > <name>fetcher.queue.mode</name> > > <value>byHost</value> > > <description>Determines how to put URLs into queues. Default value is > > 'byHost', > > also takes 'byDomain' or 'byIP'. > > </description> > > </property> > > > > <property> > > <name>fetcher.verbose</name> > > <value>false</value> > > <description>If true, fetcher will log more verbosely.</description> > > </property> > > > > > > I inherited this and not that well versed on nutch. Many hours of > > searching, and trying what I have found but still no luck. Can't get it to > > just search the local > > system http://127.0.0.1:8080/cocoon > > > > > > And help would be greatly appreciated. > > > > -- > > Mitch Baker <[email protected]> > > LSA > > -- Mitch Baker <[email protected]> LSA
