I apologize, but I haven't found much Nutch documentation that deals with the user-agent and robots.txt. Why am I being blocked when the user-agent I'm sending matches the user-agent in that robots.txt?
Chip -----Original Message----- From: Markus Jelsma [mailto:markus.jel...@openindex.io] Sent: Friday, September 30, 2011 6:28 PM To: user@nutch.apache.org Cc: Chip Calhoun Subject: Re: What could be blocking me, if not robots.txt? > I've been able to run the ParserChecker now, but I'm not sure how to > understand the results. Here's what I got: > > # bin/nutch org.apache.nutch.parse.ParserChecker > http://digital.lib.umd.edu/oclc/MdU.ead.histms.0094.xml --------- Url > --------------- > http://digital.lib.umd.edu/oclc/MdU.ead.histms.0094.xml--------- > ParseData > --------- > Version: 5 > Status: success(1,0) > Title: > Outlinks: 1 > outlink: toUrl: GR:32:A:128 anchor: > Content Metadata: ETag="1fa962a-56f20-485df79c50980" Date=Fri, 30 Sep > 2011 > 19:54:14 GMT Content-Length=356128 Last-Modified=Wed, 05 May 2010 > 21:26:14 GMT Content-Type=text/xml Connection=close > Accept-Ranges=bytes > Server=Apache/2.2.3 (Red Hat) Parse Metadata: > Content-Type=application/xml This means almost everything is good to go but... > > Curl also retrieves this file, and yet I can't get my crawl to pick it up. > > Could it be an issue with robots.txt? The robots file for this site > reads as follows: User-agent: PHFAWS/Nutch-1.3 > Disallow: > > User-agent: archive.org_bot > Disallow: > > User-agent: * > Disallow: / This is the problem. > > That first user-agent is, as near as I can tell, what I'm sending. My > log shows the following: 2011-09-30 15:54:17,712 INFO http.Http - > http.agent = PHFAWS/Nutch-1.3 (American Institute of Physics: Physics > History Finding Aids Web Site; > http://www.aip.org/history/nbl/findingaids.html; > ccalh...@aip.org) > > Can anyone tell what I'm missing? Thanks. > > Chip > > > -----Original Message----- > From: Chip Calhoun [mailto:ccalh...@aip.org] > Sent: Thursday, September 29, 2011 4:12 PM > To: user@nutch.apache.org > Subject: RE: What could be blocking me, if not robots.txt? > > Ah, sorry. I had already deleted the local copy from my server > (aip.org) to avoid clutter. So yeah, that will definitely 404 now. > > Curl retrieves the whole file with no problems. I can't try the > ParserChecker today as I'm stuck away from my own machine, but I will > try it tomorrow. The fact that I can curl it at least tells me this is > a problem I need to fix in Nutch. > > Chip > > ________________________________________ > From: Markus Jelsma [markus.jel...@openindex.io] > Sent: Thursday, September 29, 2011 1:01 PM > To: user@nutch.apache.org > Cc: Chip Calhoun > Subject: Re: What could be blocking me, if not robots.txt? > > Oh, it's a 404. That makes sense. > > > Hi everyone, > > > > I'm using Nutch to crawl a few friendly sites, and am having trouble > > with some of them. One site in particular has created an exception > > for me in its robots.txt, and yet I can't crawl any of its pages. > > I've tried copying the files I want to index (3 XML documents) to my > > own server and crawling that, and it works fine that way; so > > something is keeping me from indexing any files on this other site. > > > > I compared the logs of my attempt to crawl the friendly site with my > > attempt to crawl my own site, and I've found few differences. Most > > differences come from the fact that my own site requires a > > crawlDelay, so there are many log sections along the lines of: > > > > 2011-09-29 10:57:37,529 INFO fetcher.Fetcher - -activeThreads=10, > > spinWaiting=10, fetchQueues.totalSize=2 2011-09-29 10:57:37,529 INFO > > fetcher.Fetcher - * queue: http://www.aip.org 2011-09-29 > > 10:57:37,529 INFO > > > > fetcher.Fetcher - maxThreads = 1 2011-09-29 10:57:37,529 INFO > > > > fetcher.Fetcher - inProgress = 0 2011-09-29 10:57:37,529 INFO > > fetcher.Fetcher - crawlDelay = 5000 2011-09-29 10:57:37,529 INFO > > fetcher.Fetcher - minCrawlDelay = 0 2011-09-29 10:57:37,529 INFO > > fetcher.Fetcher - nextFetchTime = 1317308262122 2011-09-29 10:57:37,529 > > INFO fetcher.Fetcher - now = 1317308257529 2011-09-29 > > 10:57:37,529 INFO fetcher.Fetcher - 0. > > http://www.aip.org/history/ead/umd/MdU.ead.histms.0067.xml 2011-09-29 > > 10:57:37,529 INFO fetcher.Fetcher - 1. > > http://www.aip.org/history/ead/umd/MdU.ead.histms.0312.xml > > > > That strikes me as probably irrelevant, but I figured I should > > mention it. The main difference I see in the logs is that the crawl > > of my own site (the crawl that worked) has the following two lines > > which do not appear in the log of my failed crawl: > > > > 2011-09-29 10:57:50,497 INFO parse.ParserFactory - The parsing plugins: > > [org.apache.nutch.parse.tika.TikaParser] are enabled via the > > plugin.includes system property, and all claim to support the > > content type application/xml, but they are not mapped to it in the > > parse-plugins.xml file 2011-09-29 10:58:23,559 INFO > > crawl.SignatureFactory - Using Signature impl: > > org.apache.nutch.crawl.MD5Signature > > > > Also, while my successful crawl has three lines like the following, > > my failed one only has two: > > > > 2011-09-29 10:58:44,824 WARN regex.RegexURLNormalizer - can't find > > rules for scope 'crawldb', using default > > > > Can anyone think of something I might have missed? > > > > Chip > > -----Original Message----- > From: Markus Jelsma [mailto:markus.jel...@openindex.io] > Sent: Thursday, September 29, 2011 1:00 PM > To: user@nutch.apache.org > Cc: Chip Calhoun > Subject: Re: What could be blocking me, if not robots.txt? > > > Hi everyone, > > > > I'm using Nutch to crawl a few friendly sites, and am having trouble > > with some of them. One site in particular has created an exception > > for me in its robots.txt, and yet I can't crawl any of its pages. > > I've tried copying the files I want to index (3 XML documents) to my > > own server and crawling that, and it works fine that way; so > > something is keeping me from indexing any files on this other site. > > > > I compared the logs of my attempt to crawl the friendly site with my > > attempt to crawl my own site, and I've found few differences. Most > > differences come from the fact that my own site requires a > > crawlDelay, so there are many log sections along the lines of: > > > > 2011-09-29 10:57:37,529 INFO fetcher.Fetcher - -activeThreads=10, > > spinWaiting=10, fetchQueues.totalSize=2 2011-09-29 10:57:37,529 INFO > > fetcher.Fetcher - * queue: http://www.aip.org 2011-09-29 > > 10:57:37,529 INFO > > > > fetcher.Fetcher - maxThreads = 1 2011-09-29 10:57:37,529 INFO > > > > fetcher.Fetcher - inProgress = 0 2011-09-29 10:57:37,529 INFO > > fetcher.Fetcher - crawlDelay = 5000 2011-09-29 10:57:37,529 INFO > > fetcher.Fetcher - minCrawlDelay = 0 2011-09-29 10:57:37,529 INFO > > fetcher.Fetcher - nextFetchTime = 1317308262122 2011-09-29 10:57:37,529 > > INFO fetcher.Fetcher - now = 1317308257529 2011-09-29 > > 10:57:37,529 INFO fetcher.Fetcher - 0. > > http://www.aip.org/history/ead/umd/MdU.ead.histms.0067.xml 2011-09-29 > > 10:57:37,529 INFO fetcher.Fetcher - 1. > > http://www.aip.org/history/ead/umd/MdU.ead.histms.0312.xml > > > > That strikes me as probably irrelevant, but I figured I should > > mention it. The main difference I see in the logs is that the crawl > > of my own site (the crawl that worked) has the following two lines > > which do not appear in the log of my failed crawl: > > > > 2011-09-29 10:57:50,497 INFO parse.ParserFactory - The parsing plugins: > > [org.apache.nutch.parse.tika.TikaParser] are enabled via the > > plugin.includes system property, and all claim to support the > > content type application/xml, but they are not mapped to it in the > > parse-plugins.xml file 2011-09-29 10:58:23,559 INFO > > crawl.SignatureFactory - Using Signature impl: > > org.apache.nutch.crawl.MD5Signature > > If this doesn't popup when crawling the site that means its not > fetched (properly). Can you try using the parser checker to download > it? Can you you curl? The fetcher should throw an exception if there's > trouble but it may also be stopped by the http.content.limit setting. > > > Also, while my successful crawl has three lines like the following, > > my failed one only has two: > > > > 2011-09-29 10:58:44,824 WARN regex.RegexURLNormalizer - can't find > > rules for scope 'crawldb', using default > > > > Can anyone think of something I might have missed? > > > > Chip
