Okay, and what does that mean? How can I repair the error?
2011/7/12 Markus Jelsma <[email protected]>: > I don't see this segment 20110712114256 being parsed. > > On Tuesday 12 July 2011 13:38:35 Paul van Hoven wrote: >> I'm not if I did understand you correct. Here is the complete output >> of my crawl: >> >> >> tom:bin toom$ ./nutch crawl /Users/toom/Downloads/nutch-1.3/crawled >> -dir /Users/toom/Downloads/nutch-1.3/sites -depth 3 -topN 50 >> solrUrl is not set, indexing will be skipped... >> crawl started in: /Users/toom/Downloads/nutch-1.3/sites >> rootUrlDir = /Users/toom/Downloads/nutch-1.3/crawled >> threads = 10 >> depth = 3 >> solrUrl=null >> topN = 50 >> Injector: starting at 2011-07-12 12:28:49 >> Injector: crawlDb: /Users/toom/Downloads/nutch-1.3/sites/crawldb >> Injector: urlDir: /Users/toom/Downloads/nutch-1.3/crawled >> Injector: Converting injected urls to crawl db entries. >> Injector: Merging injected urls into crawl db. >> Injector: finished at 2011-07-12 12:28:53, elapsed: 00:00:04 >> Generator: starting at 2011-07-12 12:28:53 >> Generator: Selecting best-scoring urls due for fetch. >> Generator: filtering: true >> Generator: normalizing: true >> Generator: topN: 50 >> Generator: jobtracker is 'local', generating exactly one partition. >> Generator: Partitioning selected urls for politeness. >> Generator: segment: >> /Users/toom/Downloads/nutch-1.3/sites/segments/20110712122856 >> Generator: finished at 2011-07-12 12:28:57, elapsed: 00:00:04 >> Fetcher: Your 'http.agent.name' value should be listed first in >> 'http.robots.agents' property. >> Fetcher: starting at 2011-07-12 12:28:57 >> Fetcher: segment: >> /Users/toom/Downloads/nutch-1.3/sites/segments/20110712122856 Fetcher: >> threads: 10 >> QueueFeeder finished: total 1 records + hit by time limit :0 >> fetching http://nutch.apache.org/ >> -finishing thread FetcherThread, activeThreads=1 >> -finishing thread FetcherThread, activeThreads=1 >> -finishing thread FetcherThread, activeThreads=1 >> -finishing thread FetcherThread, activeThreads=1 >> -finishing thread FetcherThread, activeThreads=1 >> -finishing thread FetcherThread, activeThreads=1 >> -finishing thread FetcherThread, activeThreads=1 >> -finishing thread FetcherThread, activeThreads=1 >> -finishing thread FetcherThread, activeThreads=1 >> -activeThreads=1, spinWaiting=0, fetchQueues.totalSize=0 >> -finishing thread FetcherThread, activeThreads=0 >> -activeThreads=0, spinWaiting=0, fetchQueues.totalSize=0 >> -activeThreads=0 >> Fetcher: finished at 2011-07-12 12:29:01, elapsed: 00:00:03 >> ParseSegment: starting at 2011-07-12 12:29:01 >> ParseSegment: segment: >> /Users/toom/Downloads/nutch-1.3/sites/segments/20110712122856 >> ParseSegment: finished at 2011-07-12 12:29:03, elapsed: 00:00:02 >> CrawlDb update: starting at 2011-07-12 12:29:03 >> CrawlDb update: db: /Users/toom/Downloads/nutch-1.3/sites/crawldb >> CrawlDb update: segments: >> [/Users/toom/Downloads/nutch-1.3/sites/segments/20110712122856] >> CrawlDb update: additions allowed: true >> CrawlDb update: URL normalizing: true >> CrawlDb update: URL filtering: true >> CrawlDb update: Merging segment data into db. >> CrawlDb update: finished at 2011-07-12 12:29:06, elapsed: 00:00:02 >> Generator: starting at 2011-07-12 12:29:06 >> Generator: Selecting best-scoring urls due for fetch. >> Generator: filtering: true >> Generator: normalizing: true >> Generator: topN: 50 >> Generator: jobtracker is 'local', generating exactly one partition. >> Generator: Partitioning selected urls for politeness. >> Generator: segment: >> /Users/toom/Downloads/nutch-1.3/sites/segments/20110712122908 >> Generator: finished at 2011-07-12 12:29:10, elapsed: 00:00:03 >> Fetcher: Your 'http.agent.name' value should be listed first in >> 'http.robots.agents' property. >> Fetcher: starting at 2011-07-12 12:29:10 >> Fetcher: segment: >> /Users/toom/Downloads/nutch-1.3/sites/segments/20110712122908 Fetcher: >> threads: 10 >> QueueFeeder finished: total 50 records + hit by time limit :0 >> fetching http://www.cafepress.com/nutch/ >> fetching http://creativecommons.org/press-releases/entry/5064 >> fetching http://blog.foofactory.fi/2007/03/twice-speed-half-size.html >> fetching http://www.apache.org/dist/nutch/CHANGES-1.0.txt >> fetching http://eu.apachecon.com/c/aceu2009/sessions/138 >> fetching http://www.us.apachecon.com/c/acus2009/ >> fetching http://issues.apache.org/jira/browse/NUTCH >> fetching http://forrest.apache.org/ >> fetching http://hadoop.apache.org/ >> fetching http://wiki.apache.org/nutch/ >> fetching http://nutch.apache.org/credits.html >> fetching http://tika.apache.org/ >> fetching http://lucene.apache.org/solr/ >> fetching http://osuosl.org/news_folder/nutch >> fetching http://www.eu.apachecon.com/c/aceu2009/ >> -activeThreads=10, spinWaiting=1, fetchQueues.totalSize=35 >> -activeThreads=10, spinWaiting=8, fetchQueues.totalSize=35 >> -activeThreads=10, spinWaiting=10, fetchQueues.totalSize=35 >> -activeThreads=10, spinWaiting=10, fetchQueues.totalSize=35 >> -activeThreads=10, spinWaiting=10, fetchQueues.totalSize=35 >> fetching http://www.apache.org/ >> fetching http://eu.apachecon.com/c/aceu2009/sessions/251 >> fetching http://nutch.apache.org/skin/fontsize.js >> -activeThreads=10, spinWaiting=8, fetchQueues.totalSize=32 >> fetching http://www.us.apachecon.com/c/acus2009/schedule >> fetching http://wiki.apache.org/nutch/NutchTutorial >> -activeThreads=10, spinWaiting=8, fetchQueues.totalSize=30 >> fetching http://lucene.apache.org/java/ >> -activeThreads=10, spinWaiting=10, fetchQueues.totalSize=29 >> -activeThreads=10, spinWaiting=10, fetchQueues.totalSize=29 >> fetching http://www.apache.org/dyn/closer.cgi/nutch/ >> -activeThreads=10, spinWaiting=9, fetchQueues.totalSize=28 >> -activeThreads=10, spinWaiting=10, fetchQueues.totalSize=28 >> fetching http://eu.apachecon.com/c/aceu2009/sessions/197 >> fetching http://nutch.apache.org/nightly.html >> -activeThreads=10, spinWaiting=10, fetchQueues.totalSize=26 >> fetching http://wiki.apache.org/nutch/FAQ >> -activeThreads=10, spinWaiting=9, fetchQueues.totalSize=25 >> -activeThreads=10, spinWaiting=9, fetchQueues.totalSize=25 >> fetching http://www.apache.org/licenses/ >> -activeThreads=10, spinWaiting=9, fetchQueues.totalSize=24 >> -activeThreads=10, spinWaiting=10, fetchQueues.totalSize=24 >> fetching http://eu.apachecon.com/c/aceu2009/sessions/136 >> fetching http://nutch.apache.org/apidocs-1.3/index.html >> -activeThreads=10, spinWaiting=8, fetchQueues.totalSize=22 >> -activeThreads=10, spinWaiting=10, fetchQueues.totalSize=22 >> -activeThreads=10, spinWaiting=10, fetchQueues.totalSize=22 >> fetching http://www.apache.org/dist/nutch/CHANGES-1.2.txt >> -activeThreads=10, spinWaiting=9, fetchQueues.totalSize=21 >> -activeThreads=10, spinWaiting=9, fetchQueues.totalSize=21 >> -activeThreads=10, spinWaiting=10, fetchQueues.totalSize=21 >> fetching http://nutch.apache.org/skin/breadcrumbs.js >> fetching http://eu.apachecon.com/c/aceu2009/sessions/165 >> -activeThreads=10, spinWaiting=9, fetchQueues.totalSize=19 >> -activeThreads=10, spinWaiting=10, fetchQueues.totalSize=19 >> -activeThreads=10, spinWaiting=10, fetchQueues.totalSize=19 >> fetching http://www.apache.org/dist/nutch/CHANGES-0.9.txt >> -activeThreads=10, spinWaiting=10, fetchQueues.totalSize=18 >> -activeThreads=10, spinWaiting=10, fetchQueues.totalSize=18 >> fetching http://eu.apachecon.com/c/aceu2009/sessions/201 >> -activeThreads=10, spinWaiting=9, fetchQueues.totalSize=17 >> fetching http://nutch.apache.org/skin/getMenu.js >> -activeThreads=10, spinWaiting=10, fetchQueues.totalSize=16 >> fetching http://www.apache.org/dist/nutch/CHANGES-1.1.txt >> -activeThreads=10, spinWaiting=9, fetchQueues.totalSize=15 >> -activeThreads=10, spinWaiting=9, fetchQueues.totalSize=15 >> -activeThreads=10, spinWaiting=10, fetchQueues.totalSize=15 >> -activeThreads=10, spinWaiting=9, fetchQueues.totalSize=15 >> fetching http://eu.apachecon.com/c/aceu2009/sessions/137 >> fetching http://nutch.apache.org/index.html >> -activeThreads=10, spinWaiting=9, fetchQueues.totalSize=13 >> fetching http://www.apache.org/dist/nutch/CHANGES-0.8.1.txt >> -activeThreads=10, spinWaiting=10, fetchQueues.totalSize=12 >> -activeThreads=10, spinWaiting=10, fetchQueues.totalSize=12 >> -activeThreads=10, spinWaiting=10, fetchQueues.totalSize=12 >> -activeThreads=10, spinWaiting=10, fetchQueues.totalSize=12 >> fetching >> http://www.apache.org/foundation/records/minutes/2010/board_minutes_2010_0 >> 4_21.txt fetching http://eu.apachecon.com/c/aceu2009/sessions/250 >> -activeThreads=10, spinWaiting=8, fetchQueues.totalSize=10 >> fetching http://nutch.apache.org/mailing_lists.html >> -activeThreads=10, spinWaiting=10, fetchQueues.totalSize=9 >> -activeThreads=10, spinWaiting=10, fetchQueues.totalSize=9 >> -activeThreads=10, spinWaiting=10, fetchQueues.totalSize=9 >> -activeThreads=10, spinWaiting=10, fetchQueues.totalSize=9 >> fetching http://www.apache.org/dist/nutch/CHANGES-1.3.txt >> -activeThreads=10, spinWaiting=9, fetchQueues.totalSize=8 >> fetching http://nutch.apache.org/bot.html >> -activeThreads=10, spinWaiting=9, fetchQueues.totalSize=7 >> -activeThreads=10, spinWaiting=10, fetchQueues.totalSize=7 >> -activeThreads=10, spinWaiting=10, fetchQueues.totalSize=7 >> -activeThreads=10, spinWaiting=10, fetchQueues.totalSize=7 >> -activeThreads=10, spinWaiting=10, fetchQueues.totalSize=7 >> -activeThreads=10, spinWaiting=10, fetchQueues.totalSize=7 >> fetching http://nutch.apache.org/issue_tracking.html >> -activeThreads=10, spinWaiting=10, fetchQueues.totalSize=6 >> -activeThreads=10, spinWaiting=10, fetchQueues.totalSize=6 >> -activeThreads=10, spinWaiting=10, fetchQueues.totalSize=6 >> -activeThreads=10, spinWaiting=10, fetchQueues.totalSize=6 >> -activeThreads=10, spinWaiting=10, fetchQueues.totalSize=6 >> fetching http://nutch.apache.org/about.html >> -activeThreads=10, spinWaiting=9, fetchQueues.totalSize=5 >> -activeThreads=10, spinWaiting=10, fetchQueues.totalSize=5 >> -activeThreads=10, spinWaiting=10, fetchQueues.totalSize=5 >> -activeThreads=10, spinWaiting=10, fetchQueues.totalSize=5 >> -activeThreads=10, spinWaiting=10, fetchQueues.totalSize=5 >> -activeThreads=10, spinWaiting=10, fetchQueues.totalSize=5 >> fetching http://nutch.apache.org/i18n.html >> -activeThreads=10, spinWaiting=10, fetchQueues.totalSize=4 >> * queue: http://nutch.apache.org >> maxThreads = 1 >> inProgress = 0 >> crawlDelay = 5000 >> minCrawlDelay = 0 >> nextFetchTime = 1310466617719 >> now = 1310466613063 >> 0. http://nutch.apache.org/version_control.html >> 1. http://nutch.apache.org/skin/getBlank.js >> 2. http://nutch.apache.org/index.pdf >> 3. http://nutch.apache.org/apidocs-1.2/index.html >> -activeThreads=10, spinWaiting=10, fetchQueues.totalSize=4 >> * queue: http://nutch.apache.org >> maxThreads = 1 >> inProgress = 0 >> crawlDelay = 5000 >> minCrawlDelay = 0 >> nextFetchTime = 1310466617719 >> now = 1310466614064 >> 0. http://nutch.apache.org/version_control.html >> 1. http://nutch.apache.org/skin/getBlank.js >> 2. http://nutch.apache.org/index.pdf >> 3. http://nutch.apache.org/apidocs-1.2/index.html >> -activeThreads=10, spinWaiting=10, fetchQueues.totalSize=4 >> * queue: http://nutch.apache.org >> maxThreads = 1 >> inProgress = 0 >> crawlDelay = 5000 >> minCrawlDelay = 0 >> nextFetchTime = 1310466617719 >> now = 1310466615066 >> 0. http://nutch.apache.org/version_control.html >> 1. http://nutch.apache.org/skin/getBlank.js >> 2. http://nutch.apache.org/index.pdf >> 3. http://nutch.apache.org/apidocs-1.2/index.html >> -activeThreads=10, spinWaiting=10, fetchQueues.totalSize=4 >> * queue: http://nutch.apache.org >> maxThreads = 1 >> inProgress = 0 >> crawlDelay = 5000 >> minCrawlDelay = 0 >> nextFetchTime = 1310466617719 >> now = 1310466616068 >> 0. http://nutch.apache.org/version_control.html >> 1. http://nutch.apache.org/skin/getBlank.js >> 2. http://nutch.apache.org/index.pdf >> 3. http://nutch.apache.org/apidocs-1.2/index.html >> -activeThreads=10, spinWaiting=10, fetchQueues.totalSize=4 >> * queue: http://nutch.apache.org >> maxThreads = 1 >> inProgress = 0 >> crawlDelay = 5000 >> minCrawlDelay = 0 >> nextFetchTime = 1310466617719 >> now = 1310466617069 >> 0. http://nutch.apache.org/version_control.html >> 1. http://nutch.apache.org/skin/getBlank.js >> 2. http://nutch.apache.org/index.pdf >> 3. http://nutch.apache.org/apidocs-1.2/index.html >> fetching http://nutch.apache.org/version_control.html >> -activeThreads=10, spinWaiting=9, fetchQueues.totalSize=3 >> * queue: http://nutch.apache.org >> maxThreads = 1 >> inProgress = 1 >> crawlDelay = 5000 >> minCrawlDelay = 0 >> nextFetchTime = 1310466617719 >> now = 1310466618071 >> 0. http://nutch.apache.org/skin/getBlank.js >> 1. http://nutch.apache.org/index.pdf >> 2. http://nutch.apache.org/apidocs-1.2/index.html >> -activeThreads=10, spinWaiting=10, fetchQueues.totalSize=3 >> * queue: http://nutch.apache.org >> maxThreads = 1 >> inProgress = 0 >> crawlDelay = 5000 >> minCrawlDelay = 0 >> nextFetchTime = 1310466623151 >> now = 1310466619073 >> 0. http://nutch.apache.org/skin/getBlank.js >> 1. http://nutch.apache.org/index.pdf >> 2. http://nutch.apache.org/apidocs-1.2/index.html >> -activeThreads=10, spinWaiting=10, fetchQueues.totalSize=3 >> * queue: http://nutch.apache.org >> maxThreads = 1 >> inProgress = 0 >> crawlDelay = 5000 >> minCrawlDelay = 0 >> nextFetchTime = 1310466623151 >> now = 1310466620075 >> 0. http://nutch.apache.org/skin/getBlank.js >> 1. http://nutch.apache.org/index.pdf >> 2. http://nutch.apache.org/apidocs-1.2/index.html >> -activeThreads=10, spinWaiting=10, fetchQueues.totalSize=3 >> * queue: http://nutch.apache.org >> maxThreads = 1 >> inProgress = 0 >> crawlDelay = 5000 >> minCrawlDelay = 0 >> nextFetchTime = 1310466623151 >> now = 1310466621077 >> 0. http://nutch.apache.org/skin/getBlank.js >> 1. http://nutch.apache.org/index.pdf >> 2. http://nutch.apache.org/apidocs-1.2/index.html >> -activeThreads=10, spinWaiting=10, fetchQueues.totalSize=3 >> * queue: http://nutch.apache.org >> maxThreads = 1 >> inProgress = 0 >> crawlDelay = 5000 >> minCrawlDelay = 0 >> nextFetchTime = 1310466623151 >> now = 1310466622078 >> 0. http://nutch.apache.org/skin/getBlank.js >> 1. http://nutch.apache.org/index.pdf >> 2. http://nutch.apache.org/apidocs-1.2/index.html >> -activeThreads=10, spinWaiting=10, fetchQueues.totalSize=3 >> * queue: http://nutch.apache.org >> maxThreads = 1 >> inProgress = 0 >> crawlDelay = 5000 >> minCrawlDelay = 0 >> nextFetchTime = 1310466623151 >> now = 1310466623080 >> 0. http://nutch.apache.org/skin/getBlank.js >> 1. http://nutch.apache.org/index.pdf >> 2. http://nutch.apache.org/apidocs-1.2/index.html >> fetching http://nutch.apache.org/skin/getBlank.js >> -activeThreads=10, spinWaiting=10, fetchQueues.totalSize=2 >> * queue: http://nutch.apache.org >> maxThreads = 1 >> inProgress = 0 >> crawlDelay = 5000 >> minCrawlDelay = 0 >> nextFetchTime = 1310466628578 >> now = 1310466624082 >> 0. http://nutch.apache.org/index.pdf >> 1. http://nutch.apache.org/apidocs-1.2/index.html >> -activeThreads=10, spinWaiting=10, fetchQueues.totalSize=2 >> * queue: http://nutch.apache.org >> maxThreads = 1 >> inProgress = 0 >> crawlDelay = 5000 >> minCrawlDelay = 0 >> nextFetchTime = 1310466628578 >> now = 1310466625084 >> 0. http://nutch.apache.org/index.pdf >> 1. http://nutch.apache.org/apidocs-1.2/index.html >> -activeThreads=10, spinWaiting=10, fetchQueues.totalSize=2 >> * queue: http://nutch.apache.org >> maxThreads = 1 >> inProgress = 0 >> crawlDelay = 5000 >> minCrawlDelay = 0 >> nextFetchTime = 1310466628578 >> now = 1310466626086 >> 0. http://nutch.apache.org/index.pdf >> 1. http://nutch.apache.org/apidocs-1.2/index.html >> -activeThreads=10, spinWaiting=10, fetchQueues.totalSize=2 >> * queue: http://nutch.apache.org >> maxThreads = 1 >> inProgress = 0 >> crawlDelay = 5000 >> minCrawlDelay = 0 >> nextFetchTime = 1310466628578 >> now = 1310466627088 >> 0. http://nutch.apache.org/index.pdf >> 1. http://nutch.apache.org/apidocs-1.2/index.html >> -activeThreads=10, spinWaiting=10, fetchQueues.totalSize=2 >> * queue: http://nutch.apache.org >> maxThreads = 1 >> inProgress = 0 >> crawlDelay = 5000 >> minCrawlDelay = 0 >> nextFetchTime = 1310466628578 >> now = 1310466628089 >> 0. http://nutch.apache.org/index.pdf >> 1. http://nutch.apache.org/apidocs-1.2/index.html >> fetching http://nutch.apache.org/index.pdf >> -activeThreads=10, spinWaiting=9, fetchQueues.totalSize=1 >> * queue: http://nutch.apache.org >> maxThreads = 1 >> inProgress = 1 >> crawlDelay = 5000 >> minCrawlDelay = 0 >> nextFetchTime = 1310466628578 >> now = 1310466629090 >> 0. http://nutch.apache.org/apidocs-1.2/index.html >> -activeThreads=10, spinWaiting=10, fetchQueues.totalSize=1 >> * queue: http://nutch.apache.org >> maxThreads = 1 >> inProgress = 0 >> crawlDelay = 5000 >> minCrawlDelay = 0 >> nextFetchTime = 1310466634844 >> now = 1310466630092 >> 0. http://nutch.apache.org/apidocs-1.2/index.html >> -activeThreads=10, spinWaiting=10, fetchQueues.totalSize=1 >> * queue: http://nutch.apache.org >> maxThreads = 1 >> inProgress = 0 >> crawlDelay = 5000 >> minCrawlDelay = 0 >> nextFetchTime = 1310466634844 >> now = 1310466631094 >> 0. http://nutch.apache.org/apidocs-1.2/index.html >> -activeThreads=10, spinWaiting=10, fetchQueues.totalSize=1 >> * queue: http://nutch.apache.org >> maxThreads = 1 >> inProgress = 0 >> crawlDelay = 5000 >> minCrawlDelay = 0 >> nextFetchTime = 1310466634844 >> now = 1310466632095 >> 0. http://nutch.apache.org/apidocs-1.2/index.html >> -activeThreads=10, spinWaiting=10, fetchQueues.totalSize=1 >> * queue: http://nutch.apache.org >> maxThreads = 1 >> inProgress = 0 >> crawlDelay = 5000 >> minCrawlDelay = 0 >> nextFetchTime = 1310466634844 >> now = 1310466633097 >> 0. http://nutch.apache.org/apidocs-1.2/index.html >> -activeThreads=10, spinWaiting=10, fetchQueues.totalSize=1 >> * queue: http://nutch.apache.org >> maxThreads = 1 >> inProgress = 0 >> crawlDelay = 5000 >> minCrawlDelay = 0 >> nextFetchTime = 1310466634844 >> now = 1310466634099 >> 0. http://nutch.apache.org/apidocs-1.2/index.html >> fetching http://nutch.apache.org/apidocs-1.2/index.html >> -finishing thread FetcherThread, activeThreads=9 >> -finishing thread FetcherThread, activeThreads=8 >> -finishing thread FetcherThread, activeThreads=7 >> -finishing thread FetcherThread, activeThreads=6 >> -finishing thread FetcherThread, activeThreads=5 >> -activeThreads=5, spinWaiting=4, fetchQueues.totalSize=0 >> -finishing thread FetcherThread, activeThreads=4 >> -finishing thread FetcherThread, activeThreads=3 >> -finishing thread FetcherThread, activeThreads=2 >> -finishing thread FetcherThread, activeThreads=1 >> -finishing thread FetcherThread, activeThreads=0 >> -activeThreads=0, spinWaiting=0, fetchQueues.totalSize=0 >> -activeThreads=0 >> Fetcher: finished at 2011-07-12 12:30:37, elapsed: 00:01:27 >> ParseSegment: starting at 2011-07-12 12:30:37 >> ParseSegment: segment: >> /Users/toom/Downloads/nutch-1.3/sites/segments/20110712122908 >> Error parsing: http://nutch.apache.org/skin/breadcrumbs.js: >> failed(2,0): Can't retrieve Tika parser for mime-type >> application/javascript >> Error parsing: http://nutch.apache.org/skin/fontsize.js: failed(2,0): >> Can't retrieve Tika parser for mime-type application/javascript >> Error parsing: http://nutch.apache.org/skin/getBlank.js: failed(2,0): >> Can't retrieve Tika parser for mime-type application/javascript >> Error parsing: http://nutch.apache.org/skin/getMenu.js: failed(2,0): >> Can't retrieve Tika parser for mime-type application/javascript >> ParseSegment: finished at 2011-07-12 12:30:46, elapsed: 00:00:08 >> CrawlDb update: starting at 2011-07-12 12:30:46 >> CrawlDb update: db: /Users/toom/Downloads/nutch-1.3/sites/crawldb >> CrawlDb update: segments: >> [/Users/toom/Downloads/nutch-1.3/sites/segments/20110712122908] >> CrawlDb update: additions allowed: true >> CrawlDb update: URL normalizing: true >> CrawlDb update: URL filtering: true >> CrawlDb update: Merging segment data into db. >> CrawlDb update: finished at 2011-07-12 12:30:48, elapsed: 00:00:02 >> Generator: starting at 2011-07-12 12:30:48 >> Generator: Selecting best-scoring urls due for fetch. >> Generator: filtering: true >> Generator: normalizing: true >> Generator: topN: 50 >> Generator: jobtracker is 'local', generating exactly one partition. >> Generator: Partitioning selected urls for politeness. >> Generator: segment: >> /Users/toom/Downloads/nutch-1.3/sites/segments/20110712123051 >> Generator: finished at 2011-07-12 12:30:52, elapsed: 00:00:03 >> Fetcher: Your 'http.agent.name' value should be listed first in >> 'http.robots.agents' property. >> Fetcher: starting at 2011-07-12 12:30:52 >> Fetcher: segment: >> /Users/toom/Downloads/nutch-1.3/sites/segments/20110712123051 Fetcher: >> threads: 10 >> QueueFeeder finished: total 50 records + hit by time limit :0 >> fetching http://www.onehippo.com/ >> fetching http://apacheconeu.blogspot.com/ >> fetching http://www.day.com/ >> fetching http://www.func.nl/apacheconeu2009 >> fetching http://www.thawte.com/ >> fetching http://eu.apachecon.com/c/aceu2009/about >> fetching http://www.us.apachecon.com/c/acus2009/sessions/333 >> fetching http://www.joost.com/ >> fetching http://developer.yahoo.com/blogs/hadoop/ >> fetching http://www.springsource.com/ >> fetching http://www.isi.edu/~koehn/europarl/ >> fetching http://www.topicus.nl/ >> fetching http://opensource.hp.com/ >> fetching http://nutch.apache.org/apidocs-1.3/overview-frame.html >> -activeThreads=10, spinWaiting=0, fetchQueues.totalSize=36 >> fetching http://www.haloworldwide.com/ >> fetching https://builds.apache.org/job/Nutch-trunk/javadoc/ >> fetch of https://builds.apache.org/job/Nutch-trunk/javadoc/ failed >> with: org.apache.nutch.protocol.ProtocolNotFound: protocol not found >> for url=https >> fetching http://www.hotwaxmedia.com/ >> fetching http://lucene.apache.org/hadoop >> fetching http://www.cloudera.com/ >> fetching http://code.google.com/opensource/ >> fetching http://www.lucidimagination.com/ >> fetching http://apache.lehtivihrea.org/nutch/ >> fetching http://www.eu.apachecon.com/c/aceu2009/about/meetups >> -activeThreads=10, spinWaiting=4, fetchQueues.totalSize=27 >> -activeThreads=10, spinWaiting=10, fetchQueues.totalSize=27 >> -activeThreads=10, spinWaiting=10, fetchQueues.totalSize=27 >> -activeThreads=10, spinWaiting=10, fetchQueues.totalSize=27 >> fetching http://www.us.apachecon.com/c/acus2009/sessions/334 >> -activeThreads=10, spinWaiting=9, fetchQueues.totalSize=26 >> fetching http://nutch.apache.org/apidocs-1.2/allclasses-frame.html >> fetching http://eu.apachecon.com/c/aceu2009/about/crowdvine >> -activeThreads=10, spinWaiting=8, fetchQueues.totalSize=24 >> fetching http://www.eu.apachecon.com/c/aceu2009/about/videoStreaming >> -activeThreads=10, spinWaiting=9, fetchQueues.totalSize=23 >> -activeThreads=10, spinWaiting=9, fetchQueues.totalSize=23 >> -activeThreads=10, spinWaiting=10, fetchQueues.totalSize=23 >> -activeThreads=10, spinWaiting=10, fetchQueues.totalSize=23 >> fetching http://www.us.apachecon.com/c/acus2009/sessions/335 >> fetching http://nutch.apache.org/apidocs-1.2/overview-summary.html >> -activeThreads=10, spinWaiting=9, fetchQueues.totalSize=21 >> fetching http://eu.apachecon.com/c/aceu2009/speakers >> -activeThreads=10, spinWaiting=9, fetchQueues.totalSize=20 >> -activeThreads=10, spinWaiting=9, fetchQueues.totalSize=20 >> fetching http://www.eu.apachecon.com/c/aceu2009/sponsors/sponsor >> -activeThreads=10, spinWaiting=9, fetchQueues.totalSize=19 >> -activeThreads=10, spinWaiting=9, fetchQueues.totalSize=19 >> fetching http://www.us.apachecon.com/c/acus2009/sessions/461 >> -activeThreads=10, spinWaiting=9, fetchQueues.totalSize=18 >> fetching http://nutch.apache.org/apidocs-1.3/allclasses-frame.html >> -activeThreads=10, spinWaiting=9, fetchQueues.totalSize=17 >> -activeThreads=10, spinWaiting=10, fetchQueues.totalSize=17 >> fetching http://eu.apachecon.com/c/aceu2009/articles >> -activeThreads=10, spinWaiting=9, fetchQueues.totalSize=16 >> -activeThreads=10, spinWaiting=10, fetchQueues.totalSize=16 >> -activeThreads=10, spinWaiting=10, fetchQueues.totalSize=16 >> fetching http://www.us.apachecon.com/c/acus2009/sessions/427 >> -activeThreads=10, spinWaiting=10, fetchQueues.totalSize=15 >> fetching http://nutch.apache.org/apidocs-1.2/overview-frame.html >> -activeThreads=10, spinWaiting=10, fetchQueues.totalSize=14 >> -activeThreads=10, spinWaiting=10, fetchQueues.totalSize=14 >> fetching http://eu.apachecon.com/c/aceu2009/sessions/ >> -activeThreads=10, spinWaiting=9, fetchQueues.totalSize=13 >> -activeThreads=10, spinWaiting=9, fetchQueues.totalSize=13 >> fetching http://www.us.apachecon.com/c/acus2009/sessions/430 >> -activeThreads=10, spinWaiting=9, fetchQueues.totalSize=12 >> fetching http://nutch.apache.org/apidocs-1.3/overview-summary.html >> -activeThreads=10, spinWaiting=10, fetchQueues.totalSize=11 >> -activeThreads=10, spinWaiting=10, fetchQueues.totalSize=11 >> -activeThreads=10, spinWaiting=10, fetchQueues.totalSize=11 >> -activeThreads=10, spinWaiting=10, fetchQueues.totalSize=11 >> fetching http://eu.apachecon.com/c/aceu2009/sponsors/sponsors >> -activeThreads=10, spinWaiting=9, fetchQueues.totalSize=10 >> fetching http://www.us.apachecon.com/c/acus2009/sessions/375 >> -activeThreads=10, spinWaiting=10, fetchQueues.totalSize=9 >> -activeThreads=10, spinWaiting=10, fetchQueues.totalSize=9 >> -activeThreads=10, spinWaiting=10, fetchQueues.totalSize=9 >> -activeThreads=10, spinWaiting=10, fetchQueues.totalSize=9 >> -activeThreads=10, spinWaiting=10, fetchQueues.totalSize=9 >> fetching http://eu.apachecon.com/c/ >> fetching http://www.us.apachecon.com/c/acus2009/sessions/462 >> -activeThreads=10, spinWaiting=8, fetchQueues.totalSize=7 >> -activeThreads=10, spinWaiting=10, fetchQueues.totalSize=7 >> -activeThreads=10, spinWaiting=10, fetchQueues.totalSize=7 >> -activeThreads=10, spinWaiting=10, fetchQueues.totalSize=7 >> -activeThreads=10, spinWaiting=10, fetchQueues.totalSize=7 >> -activeThreads=10, spinWaiting=10, fetchQueues.totalSize=7 >> fetching http://www.us.apachecon.com/c/acus2009/sessions/428 >> fetching http://eu.apachecon.com/c/aceu2009/schedule >> -activeThreads=10, spinWaiting=8, fetchQueues.totalSize=5 >> -activeThreads=10, spinWaiting=10, fetchQueues.totalSize=5 >> -activeThreads=10, spinWaiting=10, fetchQueues.totalSize=5 >> -activeThreads=10, spinWaiting=10, fetchQueues.totalSize=5 >> -activeThreads=10, spinWaiting=10, fetchQueues.totalSize=5 >> -activeThreads=10, spinWaiting=10, fetchQueues.totalSize=5 >> fetching http://www.us.apachecon.com/c/acus2009/sessions/331 >> fetching http://eu.apachecon.com/c/aceu2009/ >> -activeThreads=10, spinWaiting=9, fetchQueues.totalSize=3 >> * queue: http://eu.apachecon.com >> maxThreads = 1 >> inProgress = 1 >> crawlDelay = 5000 >> minCrawlDelay = 0 >> nextFetchTime = 1310466704235 >> now = 1310466704428 >> 0. http://eu.apachecon.com/js/jquery.akslideshow.js >> * queue: http://www.us.apachecon.com >> maxThreads = 1 >> inProgress = 0 >> crawlDelay = 5000 >> minCrawlDelay = 0 >> nextFetchTime = 1310466709214 >> now = 1310466704428 >> 0. http://www.us.apachecon.com/c/acus2009/sessions/437 >> 1. http://www.us.apachecon.com/c/acus2009/sessions/332 >> -activeThreads=10, spinWaiting=9, fetchQueues.totalSize=3 >> * queue: http://eu.apachecon.com >> maxThreads = 1 >> inProgress = 1 >> crawlDelay = 5000 >> minCrawlDelay = 0 >> nextFetchTime = 1310466704235 >> now = 1310466705429 >> 0. http://eu.apachecon.com/js/jquery.akslideshow.js >> * queue: http://www.us.apachecon.com >> maxThreads = 1 >> inProgress = 0 >> crawlDelay = 5000 >> minCrawlDelay = 0 >> nextFetchTime = 1310466709214 >> now = 1310466705430 >> 0. http://www.us.apachecon.com/c/acus2009/sessions/437 >> 1. http://www.us.apachecon.com/c/acus2009/sessions/332 >> -activeThreads=10, spinWaiting=10, fetchQueues.totalSize=3 >> * queue: http://eu.apachecon.com >> maxThreads = 1 >> inProgress = 0 >> crawlDelay = 5000 >> minCrawlDelay = 0 >> nextFetchTime = 1310466710968 >> now = 1310466706431 >> 0. http://eu.apachecon.com/js/jquery.akslideshow.js >> * queue: http://www.us.apachecon.com >> maxThreads = 1 >> inProgress = 0 >> crawlDelay = 5000 >> minCrawlDelay = 0 >> nextFetchTime = 1310466709214 >> now = 1310466706431 >> 0. http://www.us.apachecon.com/c/acus2009/sessions/437 >> 1. http://www.us.apachecon.com/c/acus2009/sessions/332 >> -activeThreads=10, spinWaiting=10, fetchQueues.totalSize=3 >> * queue: http://eu.apachecon.com >> maxThreads = 1 >> inProgress = 0 >> crawlDelay = 5000 >> minCrawlDelay = 0 >> nextFetchTime = 1310466710968 >> now = 1310466707433 >> 0. http://eu.apachecon.com/js/jquery.akslideshow.js >> * queue: http://www.us.apachecon.com >> maxThreads = 1 >> inProgress = 0 >> crawlDelay = 5000 >> minCrawlDelay = 0 >> nextFetchTime = 1310466709214 >> now = 1310466707433 >> 0. http://www.us.apachecon.com/c/acus2009/sessions/437 >> 1. http://www.us.apachecon.com/c/acus2009/sessions/332 >> -activeThreads=10, spinWaiting=10, fetchQueues.totalSize=3 >> * queue: http://eu.apachecon.com >> maxThreads = 1 >> inProgress = 0 >> crawlDelay = 5000 >> minCrawlDelay = 0 >> nextFetchTime = 1310466710968 >> now = 1310466708435 >> 0. http://eu.apachecon.com/js/jquery.akslideshow.js >> * queue: http://www.us.apachecon.com >> maxThreads = 1 >> inProgress = 0 >> crawlDelay = 5000 >> minCrawlDelay = 0 >> nextFetchTime = 1310466709214 >> now = 1310466708435 >> 0. http://www.us.apachecon.com/c/acus2009/sessions/437 >> 1. http://www.us.apachecon.com/c/acus2009/sessions/332 >> fetching http://www.us.apachecon.com/c/acus2009/sessions/437 >> -activeThreads=10, spinWaiting=9, fetchQueues.totalSize=2 >> * queue: http://eu.apachecon.com >> maxThreads = 1 >> inProgress = 0 >> crawlDelay = 5000 >> minCrawlDelay = 0 >> nextFetchTime = 1310466710968 >> now = 1310466709442 >> 0. http://eu.apachecon.com/js/jquery.akslideshow.js >> * queue: http://www.us.apachecon.com >> maxThreads = 1 >> inProgress = 1 >> crawlDelay = 5000 >> minCrawlDelay = 0 >> nextFetchTime = 1310466709214 >> now = 1310466709442 >> 0. http://www.us.apachecon.com/c/acus2009/sessions/332 >> -activeThreads=10, spinWaiting=10, fetchQueues.totalSize=2 >> * queue: http://eu.apachecon.com >> maxThreads = 1 >> inProgress = 0 >> crawlDelay = 5000 >> minCrawlDelay = 0 >> nextFetchTime = 1310466710968 >> now = 1310466710444 >> 0. http://eu.apachecon.com/js/jquery.akslideshow.js >> * queue: http://www.us.apachecon.com >> maxThreads = 1 >> inProgress = 0 >> crawlDelay = 5000 >> minCrawlDelay = 0 >> nextFetchTime = 1310466714813 >> now = 1310466710444 >> 0. http://www.us.apachecon.com/c/acus2009/sessions/332 >> fetching http://eu.apachecon.com/js/jquery.akslideshow.js >> -activeThreads=10, spinWaiting=10, fetchQueues.totalSize=1 >> * queue: http://www.us.apachecon.com >> maxThreads = 1 >> inProgress = 0 >> crawlDelay = 5000 >> minCrawlDelay = 0 >> nextFetchTime = 1310466714813 >> now = 1310466711446 >> 0. http://www.us.apachecon.com/c/acus2009/sessions/332 >> -activeThreads=10, spinWaiting=10, fetchQueues.totalSize=1 >> * queue: http://www.us.apachecon.com >> maxThreads = 1 >> inProgress = 0 >> crawlDelay = 5000 >> minCrawlDelay = 0 >> nextFetchTime = 1310466714813 >> now = 1310466712447 >> 0. http://www.us.apachecon.com/c/acus2009/sessions/332 >> -activeThreads=10, spinWaiting=10, fetchQueues.totalSize=1 >> * queue: http://www.us.apachecon.com >> maxThreads = 1 >> inProgress = 0 >> crawlDelay = 5000 >> minCrawlDelay = 0 >> nextFetchTime = 1310466714813 >> now = 1310466713448 >> 0. http://www.us.apachecon.com/c/acus2009/sessions/332 >> -activeThreads=10, spinWaiting=10, fetchQueues.totalSize=1 >> * queue: http://www.us.apachecon.com >> maxThreads = 1 >> inProgress = 0 >> crawlDelay = 5000 >> minCrawlDelay = 0 >> nextFetchTime = 1310466714813 >> now = 1310466714450 >> 0. http://www.us.apachecon.com/c/acus2009/sessions/332 >> fetching http://www.us.apachecon.com/c/acus2009/sessions/332 >> -finishing thread FetcherThread, activeThreads=9 >> -finishing thread FetcherThread, activeThreads=8 >> -finishing thread FetcherThread, activeThreads=7 >> -finishing thread FetcherThread, activeThreads=6 >> -finishing thread FetcherThread, activeThreads=5 >> -finishing thread FetcherThread, activeThreads=4 >> -finishing thread FetcherThread, activeThreads=3 >> -finishing thread FetcherThread, activeThreads=2 >> -finishing thread FetcherThread, activeThreads=1 >> -finishing thread FetcherThread, activeThreads=0 >> -activeThreads=0, spinWaiting=0, fetchQueues.totalSize=0 >> -activeThreads=0 >> Fetcher: finished at 2011-07-12 12:31:55, elapsed: 00:01:03 >> ParseSegment: starting at 2011-07-12 12:31:55 >> ParseSegment: segment: >> /Users/toom/Downloads/nutch-1.3/sites/segments/20110712123051 >> Error parsing: http://eu.apachecon.com/js/jquery.akslideshow.js: >> failed(2,0): Can't retrieve Tika parser for mime-type text/javascript >> ParseSegment: finished at 2011-07-12 12:31:59, elapsed: 00:00:03 >> CrawlDb update: starting at 2011-07-12 12:31:59 >> CrawlDb update: db: /Users/toom/Downloads/nutch-1.3/sites/crawldb >> CrawlDb update: segments: >> [/Users/toom/Downloads/nutch-1.3/sites/segments/20110712123051] >> CrawlDb update: additions allowed: true >> CrawlDb update: URL normalizing: true >> CrawlDb update: URL filtering: true >> CrawlDb update: Merging segment data into db. >> CrawlDb update: finished at 2011-07-12 12:32:03, elapsed: 00:00:03 >> LinkDb: starting at 2011-07-12 12:32:03 >> LinkDb: linkdb: /Users/toom/Downloads/nutch-1.3/sites/linkdb >> LinkDb: URL normalize: true >> LinkDb: URL filter: true >> LinkDb: adding segment: >> file:/Users/toom/Downloads/nutch-1.3/sites/segments/20110707140238 >> LinkDb: adding segment: >> file:/Users/toom/Downloads/nutch-1.3/sites/segments/20110712113732 >> LinkDb: adding segment: >> file:/Users/toom/Downloads/nutch-1.3/sites/segments/20110712114256 >> LinkDb: adding segment: >> file:/Users/toom/Downloads/nutch-1.3/sites/segments/20110712122856 >> LinkDb: adding segment: >> file:/Users/toom/Downloads/nutch-1.3/sites/segments/20110712122908 >> LinkDb: adding segment: >> file:/Users/toom/Downloads/nutch-1.3/sites/segments/20110712123051 >> Exception in thread "main" >> org.apache.hadoop.mapred.InvalidInputException: Input path does not >> exist: >> file:/Users/toom/Downloads/nutch-1.3/sites/segments/20110707140238/parse_d >> ata Input path does not exist: >> file:/Users/toom/Downloads/nutch-1.3/sites/segments/20110712113732/parse_da >> ta Input path does not exist: >> file:/Users/toom/Downloads/nutch-1.3/sites/segments/20110712114256/parse_da >> ta at >> org.apache.hadoop.mapred.FileInputFormat.listStatus(FileInputFormat.java:1 >> 90) at >> org.apache.hadoop.mapred.SequenceFileInputFormat.listStatus(SequenceFileIn >> putFormat.java:44) at >> org.apache.hadoop.mapred.FileInputFormat.getSplits(FileInputFormat.java:20 >> 1) at org.apache.hadoop.mapred.JobClient.writeOldSplits(JobClient.java:810) >> at >> org.apache.hadoop.mapred.JobClient.submitJobInternal(JobClient.java:781) >> at org.apache.hadoop.mapred.JobClient.submitJob(JobClient.java:730) at >> org.apache.hadoop.mapred.JobClient.runJob(JobClient.java:1249) at >> org.apache.nutch.crawl.LinkDb.invert(LinkDb.java:175) >> at org.apache.nutch.crawl.LinkDb.invert(LinkDb.java:149) >> at org.apache.nutch.crawl.Crawl.run(Crawl.java:142) >> at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:65) >> at org.apache.nutch.crawl.Crawl.main(Crawl.java:54) >> >> 2011/7/12 Julien Nioche <[email protected]>: >> >> Actually I'm not shure if I look at the right log lines. Please >> >> explain in more detail for what exactly I should look for. Anyway I >> >> found the following line just before the error: >> >> >> >> Error parsing: http://eu.apachecon.com/js/jquery.akslideshow.js: >> >> failed(2,0): Can't retrieve Tika parser for mime-type text/javascript >> >> >> >> But I can see that parsing erros like this already appeared earlier >> >> during the crawl. >> > >> > This simply means that the javascript parser is not enabled in your conf >> > (which is the default behaviour) and as a consequence the default parser >> > (Tika) was used to try and parse it but has no resources for doing so. >> > >> > Note : we should probably add .js to the default url filters. The >> > javascript parser has been deactivated by default because it generates >> > atrocious URLs so we might as well prevent such URLs form being fetched >> > in the first place. >> > >> > Anyway this is not the source of the problem. You seem to have unparsed >> > segments among the ones specified. Could be that you interrupted a >> > previous crawl or got a problem with it and did not delete these >> > segments or the whole crawl directory. Removing the segments and calling >> > the last couple of steps manually should do the trick. >> > >> >> 2011/7/12 Markus Jelsma <[email protected]>: >> >> > Were there errors during parsing of that last segment? >> >> > >> >> >> I'm starting with nutch and I ran a simple job as described in the >> >> >> nutch tutorial. After a while I get the following error: >> >> >> >> >> >> >> >> >> CrawlDb update: URL filtering: true >> >> >> CrawlDb update: Merging segment data into db. >> >> >> CrawlDb update: finished at 2011-07-12 12:32:03, elapsed: 00:00:03 >> >> >> LinkDb: starting at 2011-07-12 12:32:03 >> >> >> LinkDb: linkdb: /Users/toom/Downloads/nutch-1.3/sites/linkdb >> >> >> LinkDb: URL normalize: true >> >> >> LinkDb: URL filter: true >> >> >> LinkDb: adding segment: >> >> >> file:/Users/toom/Downloads/nutch-1.3/sites/segments/20110707140238 >> >> >> LinkDb: adding segment: >> >> >> file:/Users/toom/Downloads/nutch-1.3/sites/segments/20110712113732 >> >> >> LinkDb: adding segment: >> >> >> file:/Users/toom/Downloads/nutch-1.3/sites/segments/20110712114256 >> >> >> LinkDb: adding segment: >> >> >> file:/Users/toom/Downloads/nutch-1.3/sites/segments/20110712122856 >> >> >> LinkDb: adding segment: >> >> >> file:/Users/toom/Downloads/nutch-1.3/sites/segments/20110712122908 >> >> >> LinkDb: adding segment: >> >> >> file:/Users/toom/Downloads/nutch-1.3/sites/segments/20110712123051 >> >> >> Exception in thread "main" >> >> >> org.apache.hadoop.mapred.InvalidInputException: Input path does not >> >> >> >> >> exist: >> >> file:/Users/toom/Downloads/nutch-1.3/sites/segments/20110707140238/parse >> >> _d >> >> >> >> >> ata Input path does not exist: >> >> file:/Users/toom/Downloads/nutch-1.3/sites/segments/20110712113732/parse >> >> _da >> >> >> >> >> ta Input path does not exist: >> >> file:/Users/toom/Downloads/nutch-1.3/sites/segments/20110712114256/parse >> >> _da >> >> >> >> >> ta at >> >> >> >> org.apache.hadoop.mapred.FileInputFormat.listStatus(FileInputFormat.java >> >> :1 >> >> >> >> >> 90) at >> >> >> >> org.apache.hadoop.mapred.SequenceFileInputFormat.listStatus(SequenceFile >> >> In >> >> >> >> >> putFormat.java:44) at >> >> >> >> org.apache.hadoop.mapred.FileInputFormat.getSplits(FileInputFormat.java: >> >> 20 >> >> >> >> >> 1) at >> >> >> >> org.apache.hadoop.mapred.JobClient.writeOldSplits(JobClient.java:810) >> >> >> >> >> at >> >> >> org.apache.hadoop.mapred.JobClient.submitJobInternal(JobClient.java:7 >> >> >> 81) at >> >> >> org.apache.hadoop.mapred.JobClient.submitJob(JobClient.java:730) at >> >> >> org.apache.hadoop.mapred.JobClient.runJob(JobClient.java:1249) at >> >> >> org.apache.nutch.crawl.LinkDb.invert(LinkDb.java:175) >> >> >> at org.apache.nutch.crawl.LinkDb.invert(LinkDb.java:149) >> >> >> at org.apache.nutch.crawl.Crawl.run(Crawl.java:142) >> >> >> at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:65) >> >> >> at org.apache.nutch.crawl.Crawl.main(Crawl.java:54) >> > >> > -- >> > * >> > *Open Source Solutions for Text Engineering >> > >> > http://digitalpebble.blogspot.com/ >> > http://www.digitalpebble.com > > -- > Markus Jelsma - CTO - Openindex > http://www.linkedin.com/in/markus17 > 050-8536620 / 06-50258350 >
