hello. I tried to crawl a certain site using both nutch 0.6 and nutch 0.7, just to compare how they are different.
However I get less urls crawled using nutch0-7 than nutch0-6. I'll paste 2 different log files below. As you can see below, both 0.6 and 0.7 fetch same number of urls in first depth, but in second depth, nutch0.7 fetches only 15 urls while nutch0.7fetches 34 urls. Of course, the configuration and settings are same. Can you tell me why I get this results? Thank you Eric Park log for Nutch 0.6 crawler --> 060328 182513 logging at INFO 060328 182513 fetching http://www.qmind.co.kr/sub4/sub4_1.htm 060328 182513 fetching http://www.qmind.co.kr/sub1/sub1_3.htm 060328 182513 fetching http://www.qmind.co.kr/sub1/sub1_5.htm 060328 182513 fetching http://www.qmind.co.kr/sub3/sub3_21.htm 060328 182513 fetching http://www.qmind.co.kr/sub1/sub1_1.htm 060328 182513 fetching http://www.qmind.co.kr/sub1/sub1_4.htm 060328 182513 fetching http://www.qmind.co.kr/sub5/sub5_4.htm 060328 182513 fetching http://www.qmind.co.kr/sub2/sub2_21.htm 060328 182513 fetching http://www.qmind.co.kr/sub6/sub6_2.htm 060328 182513 fetching http://qmind.co.kr/ 060328 182513 fetching http://www.qmind.co.kr/sub3/sub3_11.htm 060328 182514 fetching http://www.qmind.co.kr/sub2/sub2_11.htm 060328 182515 fetching http://www.qmind.co.kr/sub6/sub6_1.htm 060328 182516 fetching http://www.qmind.co.kr/sub1/sub1_2.htm 060328 182517 fetching http://www.qmind.co.kr/sub5/sub5_1.htm 060328 182528 status: segment 20060328182513, 15 pages, 0 errors, 271658 bytes, 15149 ms 060328 182528 status: 0.99016434 pages/s, 140.09691 kb/s, 18110.533bytes/page 060328 182529 Updating /usr/local/nutch-0.7/crawl_folder/crawl.qmind2/db 060328 182529 Updating for /usr/local/nutch-0.7 /crawl_folder/crawl.qmind2/segments/20060328182513 060328 182529 Processing document 0 060328 182530 Finishing update 060328 182530 Processing pagesByURL: Sorted 437 instructions in 0.013seconds. 060328 182530 Processing pagesByURL: Sorted 33615.38461538462instructions/second 060328 182530 Processing pagesByURL: Merged to new DB containing 51 records in 0.014 seconds 060328 182530 Processing pagesByURL: Merged 3642.8571428571427records/second 060328 182530 Processing pagesByMD5: Sorted 65 instructions in 0.0020seconds. 060328 182530 Processing pagesByMD5: Sorted 32500.0 instructions/second 060328 182530 Processing pagesByMD5: Merged to new DB containing 51 records in 0.0040 seconds 060328 182530 Processing pagesByMD5: Merged 12750.0 records/second 060328 182530 Processing linksByMD5: Sorted 437 instructions in 0.0060seconds. 060328 182530 Processing linksByMD5: Sorted 72833.33333333333instructions/second 060328 182530 Processing linksByMD5: Merged to new DB containing 275 records in 0.018 seconds 060328 182530 Processing linksByMD5: Merged 15277.77777777778 records/second 060328 182530 Processing linksByURL: Sorted 260 instructions in 0.0040seconds. 060328 182530 Processing linksByURL: Sorted 65000.0 instructions/second 060328 182530 Processing linksByURL: Merged to new DB containing 275 records in 0.016 seconds 060328 182530 Processing linksByURL: Merged 17187.5 records/second 060328 182530 Processing linksByMD5: Sorted 274 instructions in 0.0040seconds. 060328 182530 Processing linksByMD5: Sorted 68500.0 instructions/second 060328 182530 Processing linksByMD5: Merged to new DB containing 275 records in 0.0090 seconds 060328 182530 Processing linksByMD5: Merged 30555.55555555556 records/second 060328 182530 Update finished 060328 182530 FetchListTool started 060328 182530 Processing pagesByURL: Sorted 35 instructions in 0.0020seconds. 060328 182530 Processing pagesByURL: Sorted 17500.0 instructions/second 060328 182530 Processing pagesByURL: Merged to new DB containing 51 records in 0.0010 seconds 060328 182530 Processing pagesByURL: Merged 51000.0 records/second 060328 182530 Processing pagesByMD5: Sorted 35 instructions in 0.0020seconds. 060328 182530 Processing pagesByMD5: Sorted 17500.0 instructions/second 060328 182530 Processing pagesByMD5: Merged to new DB containing 51 records in 0.0020 seconds 060328 182530 Processing pagesByMD5: Merged 25500.0 records/second 060328 182530 Processing linksByMD5: Copied file (4096 bytes) in 0.0010secs. 060328 182530 Processing linksByURL: Copied file (4096 bytes) in 0.0020secs. 060328 182530 Processing /usr/local/nutch-0.7/crawl_folder/crawl.qmind2/segments/20060328182530/fetchlist.unsorted: Sorted 35 entries in 0.0020 seconds. 060328 182530 Processing /usr/local/nutch-0.7/crawl_folder/crawl.qmind2/segments/20060328182530/fetchlist.unsorted: Sorted 17500.0 entries/second 060328 182530 Overall processing: Sorted 35 entries in 0.0020 seconds. 060328 182530 Overall processing: Sorted 5.714285714285714E-5 entries/second 060328 182530 FetchListTool completed 060328 182530 logging at INFO 060328 182530 fetching http://www.qmind.co.kr/sub2/sub2_14.htm 060328 182530 fetching http://www.qmind.co.kr/sub2/sub2_16.htm 060328 182530 fetching http://www.qmind.co.kr/sub3/sub3_23.htm 060328 182530 fetching http://qmind.co.kr/sub1/sub1_4.htm 060328 182530 fetching http://qmind.co.kr/sub2/sub2_21.htm 060328 182530 fetching http://qmind.co.kr/sub6/sub6_2.htm 060328 182530 fetching http://www.qmind.co.kr/sub2/sub2_15.htm 060328 182530 fetching http://qmind.co.kr/sub5/sub5_4.htm 060328 182530 fetching http://qmind.co.kr/sub1/sub1_1.htm 060328 182530 fetching http://www.qmind.co.kr/sub2/sub2_12.htm 060328 182530 fetching http://www.qmind.co.kr/sub2/sub2_22.htm 060328 182531 fetching http://www.qmind.co.kr/sub2/sub2_23.htm 060328 182532 fetching http://www.qmind.co.kr/sub2/sub2_24.htm 060328 182533 fetching http://qmind.co.kr/sub5/sub5_1.htm 060328 182534 fetching http://nlp.qmind.co.kr/woori/index.jsp 060328 182534 fetch of http://nlp.qmind.co.kr/woori/index.jsp failed with: net.nutch.protocol.http.HttpError: HTTP Error: 401 060328 182534 fetching http://www.qmind.co.kr/sub3/sub3_24.htm 060328 182535 fetching http://qmind.co.kr/sub1/sub1_2.htm 060328 182536 fetching http://qmind.co.kr/sub3/sub3_21.htm 060328 182537 fetching http://qmind.co.kr/sub3/sub3_11.htm 060328 182538 fetching http://www.qmind.co.kr/sub2/sub2_13.htm 060328 182539 fetching http://www.qmind.co.kr/sub2/sub2_25.htm 060328 182540 fetching http://www.qmind.co.kr/sub3/sub3_12.htm 060328 182541 fetching http://www.qmind.co.kr/sub3/sub3_22.htm 060328 182542 fetching http://qmind.co.kr/sub1/sub1_3.htm 060328 182543 fetching http://qmind.co.kr/sub2/sub2_11.htm 060328 182544 fetching http://www.qmind.co.kr/sub2/sub2_25_1.htm 060328 182545 fetching http://www.qmind.co.kr/sub2/sub2_26.htm 060328 182546 fetching http://www.qmind.co.kr/sub3/sub3_25.htm 060328 182547 fetching http://www.qmind.co.kr/sub2/sub2_18.htm 060328 182549 fetching http://qmind.co.kr/sub1/sub1_5.htm 060328 182550 fetching http://www.qmind.co.kr/sub2/sub2_11_1.htm 060328 182551 fetching http://qmind.co.kr/sub4/sub4_1.htm 060328 182552 fetching http://www.qmind.co.kr/sub2/sub2_16_1.htm 060328 182553 fetching http://qmind.co.kr/sub6/sub6_1.htm 060328 182554 fetching http://www.qmind.co.kr/sub2/sub2_17.htm 060328 182605 status: segment 20060328182530, 34 pages, 1 errors, 707232 bytes, 35353 ms 060328 182605 status: 0.9617288 pages/s, 156.28801 kb/s, 20800.941bytes/page 060328 182606 Updating /usr/local/nutch-0.7/crawl_folder/crawl.qmind2/db 060328 182606 Updating for /usr/local/nutch-0.7 /crawl_folder/crawl.qmind2/segments/20060328182530 060328 182606 Processing document 0 060328 182606 Finishing update log for Nutch 0.7 crawler --> 060407 191934 fetching http://www.qmind.co.kr/sub4/sub4_1.htm 060407 191934 fetching http://www.qmind.co.kr/sub1/sub1_3.htm 060407 191934 fetching http://www.qmind.co.kr/sub1/sub1_4.htm 060407 191934 fetching http://www.qmind.co.kr/sub1/sub1_1.htm 060407 191934 fetching http://www.qmind.co.kr/sub3/sub3_21.htm 060407 191934 fetching http://www.qmind.co.kr/sub1/sub1_5.htm 060407 191934 fetching http://www.qmind.co.kr/sub5/sub5_4.htm 060407 191934 fetching http://www.qmind.co.kr/sub2/sub2_21.htm 060407 191934 fetching http://qmind.co.kr/ 060407 191934 fetching http://www.qmind.co.kr/sub6/sub6_2.htm 060407 191934 fetching http://www.qmind.co.kr/sub3/sub3_11.htm 060407 191935 fetching http://www.qmind.co.kr/sub2/sub2_11.htm 060407 191936 fetching http://www.qmind.co.kr/sub6/sub6_1.htm 060407 191937 fetching http://www.qmind.co.kr/sub1/sub1_2.htm 060407 191938 fetching http://www.qmind.co.kr/sub5/sub5_1.htm 060407 191949 status: segment 20060407191933, 15 pages, 0 errors, 271658 bytes, 15148 ms 060407 191949 status: 0.9902297 pages/s, 140.10617 kb/s, 18110.533bytes/page 060407 191950 Updating /home/develop/nutch-0.7.1 /crawl_folder/crawl.qmind4/db 060407 191950 Updating for /home/develop/nutch-0.7.1 /crawl_folder/crawl.qmind4/segments/20060407191933 060407 191950 Processing document 0 060407 191950 Finishing update 060407 191950 Processing pagesByURL: Sorted 359 instructions in 0.017seconds. 060407 191950 Processing pagesByURL: Sorted 21117.647058823528instructions/second 060407 191950 Processing pagesByURL: Merged to new DB containing 30 records in 0.011 seconds 060407 191950 Processing pagesByURL: Merged 2727.2727272727275records/second 060407 191950 Processing pagesByMD5: Sorted 44 instructions in 0.0070seconds. 060407 191950 Processing pagesByMD5: Sorted 6285.714285714285instructions/second 060407 191950 Processing pagesByMD5: Merged to new DB containing 30 records in 0.0050 seconds 060407 191950 Processing pagesByMD5: Merged 6000.0 records/second 060407 191950 Processing linksByMD5: Sorted 359 instructions in 0.09seconds. 060407 191950 Processing linksByMD5: Sorted 3988.888888888889instructions/second 060407 191950 Processing linksByMD5: Merged to new DB containing 254 records in 0.018 seconds 060407 191950 Processing linksByMD5: Merged 14111.111111111111records/second 060407 191950 Processing linksByURL: Sorted 239 instructions in 0.01seconds. 060407 191950 Processing linksByURL: Sorted 23900.0 instructions/second 060407 191951 Processing linksByURL: Merged to new DB containing 254 records in 0.01 seconds 060407 191951 Processing linksByURL: Merged 25400.0 records/second 060407 191951 Processing linksByMD5: Sorted 253 instructions in 0.056seconds. 060407 191951 Processing linksByMD5: Sorted 4517.857142857143instructions/second 060407 191951 Processing linksByMD5: Merged to new DB containing 254 records in 0.016 seconds 060407 191951 Processing linksByMD5: Merged 15875.0 records/second 060407 191951 Update finished 060407 191951 FetchListTool started 060407 191951 Processing pagesByURL: Sorted 14 instructions in 0.092seconds. 060407 191951 Processing pagesByURL: Sorted 152.17391304347825instructions/second 060407 191951 Processing pagesByURL: Merged to new DB containing 30 records in 0.0010 seconds 060407 191951 Processing pagesByURL: Merged 30000.0 records/second 060407 191951 Processing pagesByMD5: Sorted 14 instructions in 0.0070seconds. 060407 191951 Processing pagesByMD5: Sorted 2000.0 instructions/second 060407 191951 Processing pagesByMD5: Merged to new DB containing 30 records in 0.0020 seconds 060407 191951 Processing pagesByMD5: Merged 15000.0 records/second 060407 191951 Processing linksByMD5: Copied file (4096 bytes) in 0.0010secs. 060407 191951 Processing linksByURL: Copied file (4096 bytes) in 0.0010secs. 060407 191951 Processing /home/develop/nutch-0.7.1/crawl_folder/crawl.qmind4/segments/20060407191951/fetchlist.unsorted: Sorted 14 entries in 0.012 seconds. 060407 191951 Processing /home/develop/nutch-0.7.1/crawl_folder/crawl.qmind4/segments/20060407191951/fetchlist.unsorted: Sorted 1166.6666666666667 entries/second 060407 191951 Overall processing: Sorted 14 entries in 0.012 seconds. 060407 191951 Overall processing: Sorted 8.571428571428572E-4 entries/second 060407 191951 FetchListTool completed 060407 191951 logging at INFO 060407 191951 fetching http://qmind.co.kr/sub1/sub1_4.htm 060407 191951 fetching http://qmind.co.kr/sub2/sub2_21.htm 060407 191951 fetching http://qmind.co.kr/sub6/sub6_2.htm 060407 191951 fetching http://qmind.co.kr/sub5/sub5_4.htm 060407 191951 fetching http://qmind.co.kr/sub1/sub1_1.htm 060407 191951 fetching http://qmind.co.kr/sub5/sub5_1.htm 060407 191951 fetching http://qmind.co.kr/sub1/sub1_2.htm 060407 191951 fetching http://qmind.co.kr/sub3/sub3_21.htm 060407 191951 fetching http://qmind.co.kr/sub3/sub3_11.htm 060407 191951 fetching http://qmind.co.kr/sub1/sub1_3.htm 060407 191951 fetching http://qmind.co.kr/sub2/sub2_11.htm 060407 191952 fetching http://qmind.co.kr/sub1/sub1_5.htm 060407 191953 fetching http://qmind.co.kr/sub4/sub4_1.htm 060407 191954 fetching http://qmind.co.kr/sub6/sub6_1.htm 060407 192005 status: segment 20060407191951, 14 pages, 0 errors, 252899 bytes, 14136 ms 060407 192005 status: 0.9903792 pages/s, 139.76892 kb/s, 18064.215bytes/page 060407 192006 Updating /home/develop/nutch-0.7.1 /crawl_folder/crawl.qmind4/db 060407 192006 Updating for /home/develop/nutch-0.7.1 /crawl_folder/crawl.qmind4/segments/20060407191951 060407 192006 Processing document 0 060407 192007 Finishing update
