Many things could happen. Sample1: website was unavailable during first 10 fetches Sample2: 11th fetch used different IP, DNS-to-IP mapping changed (or may be finally resolved!) Sample3: Smth changed on a site, "redirect" added/changed, etc. Sample4: web-master modified robots.txt Sample5: big first HTML file, network errors during first 10 fetch attempts, etc.
It should be very uncommon behaviour, but it may happen... -----Original Message----- From: Bryan Woliner I am using Nutch 0.7.1 (no mapreduce) and did a whole-web crawl with 14 rounds of fetching and an urls files with one URL in it. No urls were fetched during the first 10 rounds, but then in the 11th round one URL was fetched and increasing more URLs were fetched in rounds 12-14. I am basing the numbers of URLs fetched on the output from calling bin/nutch segread (included below). I don't understand how this can happen. If a URL is not fetched during a round are its outlinks still added to the database for the next round of fetching? Why would I have 10 rounds of fetching with no URLs fetched and then suddenly have one fetched successfully in the 11th round? Any suggestions are appreciated. -Bryan Here is the output when I call: bin/nutch segread -list -dir segments run java in /usr/local/j2sdk1.4.2_08 060115 205601 parsing file:/home/bryan/nutch-0.7.1/conf/nutch-default.xml 060115 205601 parsing file:/home/bryan/nutch-0.7.1/conf/nutch-site.xml 060115 205601 No FS indicated, using default:local 060115 205601 PARSED? STARTED FINISHED COUNT DIR NAME 060115 205601 true 19691231-18:00:00 19691231-18:00:00 0 ../segments/20060115173409 060115 205601 true 19691231-18:00:00 19691231-18:00:00 0 ../segments/20060115173413 060115 205601 true 19691231-18:00:00 19691231-18:00:00 0 ../segments/20060115173417 060115 205601 true 19691231-18:00:00 19691231-18:00:00 0 ../segments/20060115173421 060115 205601 true 19691231-18:00:00 19691231-18:00:00 0 ../segments/20060115173424 060115 205601 true 19691231-18:00:00 19691231-18:00:00 0 ../segments/20060115173428 060115 205601 true 19691231-18:00:00 19691231-18:00:00 0 ../segments/20060115173432 060115 205601 true 19691231-18:00:00 19691231-18:00:00 0 ../segments/20060115173436 060115 205601 true 19691231-18:00:00 19691231-18:00:00 0 ../segments/20060115173440 060115 205601 true 19691231-18:00:00 19691231-18:00:00 0 ../segments/20060115173443 060115 205602 true 20060115-17:34:51 20060115-17:34:51 1 ../segments/20060115173447 060115 205602 true 20060115-17:34:57 20060115-17:41:07 42 ../segments/20060115173454 060115 205602 true 20060115-17:41:16 20060115-18:12:28 234 ../segments/20060115174113 060115 205602 true 20060115-18:12:37 20060115-19:51:07 738 ../segments/20060115181234 060115 205602 TOTAL: 1015 entries in 14 segments.
