Bugs item #983092, was opened at 2004-07-01 01:41 Message generated for change (Tracker Item Submitted) made by Item Submitter You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=491356&aid=983092&group_id=59548
Category: web db Group: None Status: Open Resolution: None Priority: 5 Submitted By: Fabio Gasparetti (magnum74) Assigned to: Michael Cafarella (mike_cafarella) Summary: "retry never" and "new contents" for the same resource Initial Comment: Sometimes happens that when I run the CrawlTool the update db tool shows two messages related to the same resource: "retry never" and "new contents", as if there were 2 fetch list entries. Even if the resource is available, in the Nutch db its nextFetchTime is set to Long.MAX_VALUE (i.e., never refetch). I've attached the log where I made a dump of the Nutch db before and after the fetching-updating phase. In this case the problem is with the resource: http://www.google.com/advanced_search?hl=en Fabio ... (5) refetch:true nextFetchTime:1088636344622 retriesSinceFetch:0 links:0 md5:5d365f9fac8463c63dca5a155ed51fa3 urlmesh:5d365f9fac8463c63dca5a155ed51fa3 url:http://www.google.com/advanced_search?hl=en <<<<<<<<<<<<<<<<<<--------------- .... 040630 145906 fetching http://www.google.com/about.html 040630 145907 fetching http://www.google.com/options/index.html 040630 145908 fetching http://www.google.com/imghp?hl=en&tab=wi&ie=UTF-8 040630 145909 fetching http://www.google.com/nwshp?hl=en&tab=wn&ie=UTF-8 040630 145910 fetching http://www.google.com/froogle?hl=en&tab=wf&ie=UTF-8 040630 145911 fetching http://www.google.com/preferences?hl=en 040630 145912 fetching http://www.google.com/ads/ 040630 145914 fetching http://www.google.com/grphp?hl=en&tab=wg&ie=UTF-8 040630 145915 fetching http://www.google.com/advanced_search?hl=en <<<<<<<<<<<<<<<<<<--------------- 040630 145916 fetching http://www.google.com/language_tools?hl=en 040630 145917 fetching http://www.google.com/services/ 040630 145919 status: 11 pages, 0 errors, 171063 bytes, 12141 ms 040630 145919 status: 0.90602094 pages/s, 110.07575 kb/s, 15551.182 bytes/page 040630 145919 Updating crawl.test/db 040630 145919 Updating for crawl.test/segments/20040630145906040630 040630 145919 Processing http://www.google.com/about.html 040630 145919 new contents 040630 145919 Processing http://www.google.com/options/index.html 040630 145919 new contents 040630 145919 Processing http://www.google.com/imghp?hl=en&tab=wi&ie=UTF-8 040630 145919 new contents 040630 145919 Processing http://www.google.com/nwshp?hl=en&tab=wn&ie=UTF-8 040630 145919 new contents 040630 145919 Processing http://www.google.com/froogle?hl=en&tab=wf&ie=UTF-8 040630 145919 retry never 040630 145919 Processing http://www.google.com/froogle?hl=en&tab=wf&ie=UTF-8 040630 145919 new contents 040630 145919 Processing http://www.google.com/preferences?hl=en 040630 145919 retry never 040630 145919 Processing http://www.google.com/preferences?hl=en 040630 145919 new contents 040630 145919 Processing http://www.google.com/ads/ 040630 145919 new contents 040630 145919 Processing http://www.google.com/grphp?hl=en&tab=wg&ie=UTF-8 040630 145919 new contents 040630 145919 Processing http://www.google.com/advanced_search?hl=en 040630 145919 retry never <<<<<<<<<<<<<<<<<<--------------- 040630 145919 Processing http://www.google.com/advanced_search?hl=en 040630 145919 new contents <<<<<<<<<<<<<<<<<<--------------- 040630 145919 Processing http://www.google.com/language_tools?hl=en 040630 145919 new contents 040630 145919 Processing http://www.google.com/services/ 040630 145919 new contents 040630 145919 Finishing update 040630 145919 Processing pagesByURL: Sorted 383 instructions in 0.028 seconds. 040630 145919 Processing pagesByURL: Sorted 13678.571428571428 instructions/second 040630 145919 Processing pagesByURL: Merged to new DB containing 313 records in 0.04 seconds 040630 145919 Processing pagesByURL: Merged 7825.0 records/second 040630 145919 Processing pagesByMD5: Sorted 320 instructions in 0.021 seconds. 040630 145919 Processing pagesByMD5: Sorted 15238.095238095237 instructions/second 040630 145919 Processing pagesByMD5: Merged to new DB containing 313 records in 0.024 seconds 040630 145919 Processing pagesByMD5: Merged 13041.666666666666 records/second 040630 145919 Processing linksByMD5: Sorted 377 instructions in 0.019 seconds. 040630 145919 Processing linksByMD5: Sorted 19842.105263157897 instructions/second 040630 145919 Processing linksByMD5: Merged to new DB containing 345 records in 0.024 seconds 040630 145919 Processing linksByMD5: Merged 14375.0 records/second 040630 145919 Processing linksByURL: Sorted 334 instructions in 0.017 seconds. 040630 145919 Processing linksByURL: Sorted 19647.05882352941 instructions/second 040630 145919 Processing linksByURL: Merged to new DB containing 345 records in 0.032 seconds 040630 145919 Processing linksByURL: Merged 10781.25 records/second 040630 145919 Processing linksByMD5: Sorted 342 instructions in 0.021 seconds. 040630 145919 Processing linksByMD5: Sorted 16285.714285714284 instructions/second 040630 145919 Processing linksByMD5: Merged to new DB containing 345 records in 0.022 seconds 040630 145919 Processing linksByMD5: Merged 15681.818181818182 records/second 040630 145919 Update finished ... (106) refetch:true nextFetchTime:1088636348080 retriesSinceFetch:0 links:0 md5:5ce5defa7d525e804fd7038566f6e8c0 urlmesh:5ce5defa7d525e804fd7038566f6e8c0 url:http://www.google.com/dirhp (107) refetch:false nextFetchTime:9223372036854775807 retriesSinceFetch:0 links:0 md5:5d365f9fac8463c63dca5a155ed51fa3 urlmesh:5d365f9fac8463c63dca5a155ed51fa3 url:http://www.google.com/advanced_search?hl=en <<<<<<<<<<<<<<<<<<--------------- (108) refetch:true nextFetchTime:1091228349002 retriesSinceFetch:0 links:11 md5:5d8eb4d2da38fa6f1a35b15743fcd7d4 urlmesh:b545a7e129570a8d376ad5c3e95fcfbf url:http://www.google.com/imghp?hl=en&tab=wi&ie=UTF-8 ... Nutch0.4, sdk1.4.2_01, RH7.1 ---------------------------------------------------------------------- You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=491356&aid=983092&group_id=59548 ------------------------------------------------------- This SF.Net email sponsored by Black Hat Briefings & Training. Attend Black Hat Briefings & Training, Las Vegas July 24-29 - digital self defense, top technical experts, no vendor pitches, unmatched networking opportunities. Visit www.blackhat.com _______________________________________________ Nutch-developers mailing list [EMAIL PROTECTED] https://lists.sourceforge.net/lists/listinfo/nutch-developers
