logging at info gives a rather long stack... here it is. the funny thing is, after the crash, i deleted the tmp-dir in the db folder (without cleaning up the segment folders), and recrawl the same site, and it worked without crashing.
060215 151338 Using URL normalizer: org.apache.nutch.net.BasicUrlNormalizer 060215 151338 Added 1 pages 060215 151338 Processing pagesByURL: Sorted 1 instructions in 0.01 seconds. 060215 151338 Processing pagesByURL: Sorted 100.0 instructions/second 060215 151338 Processing pagesByURL: Merged to new DB containing 1 records in 0.01 seconds 060215 151338 Processing pagesByURL: Merged 100.0 records/second 060215 151338 Processing pagesByMD5: Sorted 1 instructions in 0.0 seconds. 060215 151338 Processing pagesByMD5: Sorted Infinity instructions/second 060215 151338 Processing pagesByMD5: Merged to new DB containing 1 records in 0.0 seconds 060215 151338 Processing pagesByMD5: Merged Infinity records/second 060215 151338 Processing linksByMD5: Copied file (4096 bytes) in 0.0 secs. 060215 151338 Processing linksByURL: Copied file (4096 bytes) in 0.0 secs. 060215 151338 Processing pagesByURL: Sorted 1 instructions in 0.0 seconds. 060215 151338 Processing pagesByURL: Sorted Infinity instructions/second 060215 151338 Processing pagesByURL: Merged to new DB containing 1 records in 0.0 seconds 060215 151338 Processing pagesByURL: Merged Infinity records/second 060215 151338 Processing pagesByMD5: Sorted 1 instructions in 0.0 seconds. 060215 151338 Processing pagesByMD5: Sorted Infinity instructions/second 060215 151338 Processing pagesByMD5: Merged to new DB containing 1 records in 0.0 seconds 060215 151338 Processing pagesByMD5: Merged Infinity records/second 060215 151338 Processing linksByMD5: Copied file (4096 bytes) in 0.0 secs. 060215 151338 Processing linksByURL: Copied file (4096 bytes) in 0.0 secs. 060215 151338 Processing /var/nutch/indexes-ak/segments/20060215151338/fetchlist.unsorted: Sorted 1 entries in 0.0 seconds. 060215 151338 Processing /var/nutch/indexes-ak/segments/20060215151338/fetchlist.unsorted: Sorted Infinity entries/second 060215 151338 Overall processing: Sorted 1 entries in 0.0 seconds. 060215 151338 Overall processing: Sorted 0.0 entries/second 060215 151339 logging at INFO 060215 151339 fetching http://www.legis.state.ak.us/default.htm 060215 151339 http.proxy.host = null 060215 151339 http.proxy.port = 8080 060215 151339 http.timeout = 10000 060215 151339 http.content.limit = -1 060215 151339 http.agent = NutchCVS/0.7.1 (Nutch; http://lucene.apache.org/nutch/bot.html; [email protected]) 060215 151339 fetcher.server.delay = 1000 060215 151339 http.max.delays = 25 060215 151340 status: segment 20060215151338, 1 pages, 0 errors, 8287 bytes, 1020 ms 060215 151340 status: 0.98039216 pages/s, 63.472733 kb/s, 8287.0 bytes/page 060215 151341 Updating /var/nutch/indexes-ak/db 060215 151341 Updating for /var/nutch/indexes-ak/segments/20060215151338 060215 151341 Processing document 0 060215 151341 Finishing update 060215 151341 Processing pagesByURL: Sorted 6 instructions in 0.0 seconds. 060215 151341 Processing pagesByURL: Sorted Infinity instructions/second 060215 151341 Processing pagesByURL: Merged to new DB containing 6 records in 0.0 seconds 060215 151341 Processing pagesByURL: Merged Infinity records/second 060215 151341 Processing pagesByMD5: Sorted 7 instructions in 0.0 seconds. 060215 151341 Processing pagesByMD5: Sorted Infinity instructions/second 060215 151341 Processing pagesByMD5: Merged to new DB containing 6 records in 0.0 seconds 060215 151341 Processing pagesByMD5: Merged Infinity records/second 060215 151341 Processing linksByMD5: Sorted 6 instructions in 0.0 seconds. 060215 151341 Processing linksByMD5: Sorted Infinity instructions/second 060215 151341 Processing linksByMD5: Merged to new DB containing 5 records in 0.01 seconds 060215 151341 Processing linksByMD5: Merged 500.0 records/second 060215 151341 Processing linksByURL: Sorted 5 instructions in 0.0 seconds. 060215 151341 Processing linksByURL: Sorted Infinity instructions/second 060215 151341 Processing linksByURL: Merged to new DB containing 5 records in 0.0 seconds 060215 151341 Processing linksByURL: Merged Infinity records/second 060215 151341 Processing linksByMD5: Sorted 5 instructions in 0.0 seconds. 060215 151341 Processing linksByMD5: Sorted Infinity instructions/second 060215 151341 Processing linksByMD5: Merged to new DB containing 5 records in 0.0 seconds 060215 151341 Processing linksByMD5: Merged Infinity records/second 060215 151341 Update finished 060215 151341 Processing pagesByURL: Sorted 5 instructions in 0.0 seconds. 060215 151341 Processing pagesByURL: Sorted Infinity instructions/second 060215 151341 Processing pagesByURL: Merged to new DB containing 6 records in 0.01 seconds 060215 151341 Processing pagesByURL: Merged 600.0 records/second 060215 151341 Processing pagesByMD5: Sorted 5 instructions in 0.0 seconds. 060215 151341 Processing pagesByMD5: Sorted Infinity instructions/second 060215 151341 Processing pagesByMD5: Merged to new DB containing 6 records in 0.0 seconds 060215 151341 Processing pagesByMD5: Merged Infinity records/second 060215 151341 Processing linksByMD5: Copied file (4096 bytes) in 0.0 secs. 060215 151341 Processing linksByURL: Copied file (4096 bytes) in 0.0 secs. 060215 151341 Processing /var/nutch/indexes-ak/segments/20060215151341/fetchlist.unsorted: Sorted 5 entries in 0.0 seconds. 060215 151341 Processing /var/nutch/indexes-ak/segments/20060215151341/fetchlist.unsorted: Sorted Infinity entries/second 060215 151341 Overall processing: Sorted 5 entries in 0.0 seconds. 060215 151341 Overall processing: Sorted 0.0 entries/second 060215 151341 logging at INFO 060215 151341 fetching http://www.legis.state.ak.us/infolist.htm 060215 151341 fetching http://www.legis.state.ak.us/home.htm 060215 151341 fetching http://www.legis.state.ak.us/fhelp/helpmain.htm 060215 151341 fetching http://www.legis.state.ak.us/FBframe.htm 060215 151341 fetching http://www.legis.state.ak.us/Legframe.htm 060215 151341 fetch of http://www.legis.state.ak.us/fhelp/helpmain.htmfailed with: java.lang.Exception: org.apache.nutch.protocol.R esourceGone: Blocked by robots.txt 060215 151342 status: segment 20060215151341, 4 pages, 1 errors, 16047 bytes, 1010 ms 060215 151342 status: 3.960396 pages/s, 124.12593 kb/s, 4011.75 bytes/page 060215 151343 Updating /var/nutch/indexes-ak/db 060215 151343 Updating for /var/nutch/indexes-ak/segments/20060215151341 060215 151343 Processing document 0 060215 151343 Finishing update 060215 151343 Processing pagesByURL: Sorted 10 instructions in 0.0 seconds. 060215 151343 Processing pagesByURL: Sorted Infinity instructions/second 060215 151343 Processing pagesByURL: Merged to new DB containing 9 records in 0.0 seconds 060215 151343 Processing pagesByURL: Merged Infinity records/second 060215 151343 Processing pagesByMD5: Sorted 13 instructions in 0.0 seconds. 060215 151343 Processing pagesByMD5: Sorted Infinity instructions/second 060215 151343 Processing pagesByMD5: Merged to new DB containing 9 records in 0.0 seconds 060215 151343 Processing pagesByMD5: Merged Infinity records/second 060215 151343 Processing linksByMD5: Sorted 11 instructions in 0.0 seconds. 060215 151343 Processing linksByMD5: Sorted Infinity instructions/second 060215 151343 Processing linksByMD5: Merged to new DB containing 11 records in 0.0 seconds 060215 151343 Processing linksByMD5: Merged Infinity records/second 060215 151343 Processing linksByURL: Sorted 6 instructions in 0.0 seconds. 060215 151343 Processing linksByURL: Sorted Infinity instructions/second 060215 151343 Processing linksByURL: Merged to new DB containing 9 records in 0.01 seconds 060215 151343 Processing linksByURL: Merged 900.0 records/second 060215 151343 Processing linksByMD5: Sorted 9 instructions in 0.0 seconds. 060215 151343 Processing linksByMD5: Sorted Infinity instructions/second 060215 151343 Processing linksByMD5: Merged to new DB containing 9 records in 0.0 seconds 060215 151343 Processing linksByMD5: Merged Infinity records/second 060215 151343 Update finished 060215 151343 Processing pagesByURL: Sorted 4 instructions in 0.0 seconds. 060215 151343 Processing pagesByURL: Sorted Infinity instructions/second 060215 151343 Processing pagesByURL: Merged to new DB containing 9 records in 0.0 seconds 060215 151343 Processing pagesByURL: Merged Infinity records/second 060215 151343 Processing pagesByMD5: Sorted 4 instructions in 0.0 seconds. 060215 151343 Processing pagesByMD5: Sorted Infinity instructions/second 060215 151343 Processing pagesByMD5: Merged to new DB containing 9 records in 0.0 seconds 060215 151343 Processing pagesByMD5: Merged Infinity records/second 060215 151343 Processing linksByMD5: Copied file (4096 bytes) in 0.0 secs. 060215 151343 Processing linksByURL: Copied file (4096 bytes) in 0.0 secs. 060215 151343 Processing /var/nutch/indexes-ak/segments/20060215151343/fetchlist.unsorted: Sorted 4 entries in 0.0 seconds. 060215 151343 Processing /var/nutch/indexes-ak/segments/20060215151343/fetchlist.unsorted: Sorted Infinity entries/second 060215 151343 Overall processing: Sorted 4 entries in 0.0 seconds. 060215 151343 Overall processing: Sorted 0.0 entries/second 060215 151343 logging at INFO 060215 151343 fetching http://www.legis.state.ak.us/ 060215 151343 fetching http://www.legis.state.ak.us/folhome.htm 060215 151343 fetching http://www.legis.state.ak.us/poms/ 060215 151343 fetch of http://www.legis.state.ak.us/poms/ failed with: java.lang.Exception: org.apache.nutch.protocol.ResourceGone: Blocked by robots.txt 060215 151343 fetching http://www.legis.state.ak.us/basis/start.asp 060215 151343 fetch okay, but can't parse http://www.legis.state.ak.us/basis/start.asp, reason: failed(2,200): sun.io.MalformedInput Exception: Missing byte-order mark 060215 151344 status: segment 20060215151343, 3 pages, 1 errors, 23109 bytes, 1010 ms 060215 151344 status: 2.970297 pages/s, 178.75154 kb/s, 7703.0 bytes/page 060215 151345 Updating /var/nutch/indexes-ak/db 060215 151345 Updating for /var/nutch/indexes-ak/segments/20060215151343 060215 151345 Processing document 0 060215 151345 Finishing update 060215 151345 Processing pagesByURL: Sorted 5 instructions in 0.0 seconds. 060215 151345 Processing pagesByURL: Sorted Infinity instructions/second 060215 151345 Processing pagesByURL: Merged to new DB containing 9 records in 0.0 seconds 060215 151345 Processing pagesByURL: Merged Infinity records/second 060215 151345 Processing pagesByMD5: Sorted 8 instructions in 0.0 seconds. 060215 151345 Processing pagesByMD5: Sorted Infinity instructions/second 060215 151345 Processing pagesByMD5: Merged to new DB containing 9 records in 0.0 seconds 060215 151345 Processing pagesByMD5: Merged Infinity records/second 060215 151345 Processing linksByMD5: Sorted 13 instructions in 0.0 seconds. 060215 151345 Processing linksByMD5: Sorted Infinity instructions/second 060215 151345 Processing linksByMD5: Merged to new DB containing 14 records in 0.0 seconds 060215 151345 Processing linksByMD5: Merged Infinity records/second 060215 151345 Processing linksByURL: Sorted 9 instructions in 0.0 seconds. 060215 151345 Processing linksByURL: Sorted Infinity instructions/second 060215 151345 Processing linksByURL: Merged to new DB containing 14 records in 0.01 seconds 060215 151345 Processing linksByURL: Merged 1400.0 records/second 060215 151345 Processing linksByMD5: Sorted 10 instructions in 0.0 seconds. 060215 151345 Processing linksByMD5: Sorted Infinity instructions/second 060215 151345 Processing linksByMD5: Merged to new DB containing 14 records in 0.0 seconds 060215 151345 Processing linksByMD5: Merged Infinity records/second 060215 151345 Update finished 060215 151345 Processing pagesByURL: Sorted 1 instructions in 0.0 seconds. 060215 151345 Processing pagesByURL: Sorted Infinity instructions/second 060215 151345 Processing pagesByURL: Merged to new DB containing 9 records in 0.0 seconds 060215 151345 Processing pagesByURL: Merged Infinity records/second 060215 151345 Processing pagesByMD5: Sorted 1 instructions in 0.0 seconds. 060215 151345 Processing pagesByMD5: Sorted Infinity instructions/second 060215 151345 Processing pagesByMD5: Merged to new DB containing 9 records in 0.0 seconds 060215 151345 Processing pagesByMD5: Merged Infinity records/second 060215 151345 Processing linksByMD5: Copied file (4096 bytes) in 0.0 secs. 060215 151345 Processing linksByURL: Copied file (4096 bytes) in 0.0 secs. 060215 151345 Processing /var/nutch/indexes-ak/segments/20060215151345/fetchlist.unsorted: Sorted 1 entries in 0.0 seconds. 060215 151345 Processing /var/nutch/indexes-ak/segments/20060215151345/fetchlist.unsorted: Sorted Infinity entries/second 060215 151345 Overall processing: Sorted 1 entries in 0.0 seconds. 060215 151345 Overall processing: Sorted 0.0 entries/second 060215 151345 logging at INFO 060215 151345 fetching http://www.legis.state.ak.us/fhelp/helpmain.htm 060215 151345 fetch of http://www.legis.state.ak.us/fhelp/helpmain.htmfailed with: java.lang.Exception: org.apache.nutch.protocol.R esourceGone: Blocked by robots.txt 060215 151346 Updating /var/nutch/indexes-ak/db 060215 151346 Updating for /var/nutch/indexes-ak/segments/20060215151345 060215 151346 Processing document 0 060215 151346 Finishing update 060215 151346 Processing pagesByURL: Sorted 1 instructions in 0.0 seconds. 060215 151346 Processing pagesByURL: Sorted Infinity instructions/second 060215 151346 Processing pagesByURL: Merged to new DB containing 8 records in 0.0 seconds 060215 151346 Processing pagesByURL: Merged Infinity records/second 060215 151346 Processing pagesByMD5: Sorted 1 instructions in 0.0 seconds. 060215 151346 Processing pagesByMD5: Sorted Infinity instructions/second 060215 151346 Processing pagesByMD5: Merged to new DB containing 8 records in 0.0 seconds 060215 151346 Processing pagesByMD5: Merged Infinity records/second 060215 151346 Processing linksByMD5: Sorted 1 instructions in 0.01 seconds. 060215 151346 Processing linksByMD5: Sorted 100.0 instructions/second 060215 151346 Processing linksByMD5: Merged to new DB containing 14 records in 0.0 seconds 060215 151346 Processing linksByMD5: Merged Infinity records/second 060215 151346 Processing linksByURL: Copied file (4096 bytes) in 0.0 secs. 060215 151346 Update finished 060215 151346 Overall processing: Sorted 0 entries in 0.0 seconds. 060215 151346 Overall processing: Sorted NaN entries/second 060215 151346 Deleted empty fetchlist segment dir /var/nutch/indexes-ak/segments/20060215151346 060215 151346 Start at: 0 060215 151346 Extent: 8 060215 151346 Pages consumed: 0 (at index 0). Links fetched: 0. 060215 151346 Opened stream to file 0 060215 151346 Pages written: 0 060215 151346 Processing pagesByURL: Sorted 5 instructions in 0.0 seconds. 060215 151346 Processing pagesByURL: Sorted Infinity instructions/second 060215 151346 Processing pagesByURL: Merged to new DB containing 8 records in 0.0 seconds 060215 151346 Processing pagesByURL: Merged Infinity records/second 060215 151346 Processing pagesByMD5: Sorted 5 instructions in 0.01 seconds. 060215 151346 Processing pagesByMD5: Sorted 500.0 instructions/second 060215 151346 Processing pagesByMD5: Merged to new DB containing 8 records in 0.0 seconds 060215 151346 Processing pagesByMD5: Merged Infinity records/second 060215 151346 Processing linksByMD5: Copied file (4096 bytes) in 0.0 secs. 060215 151346 Processing linksByURL: Copied file (4096 bytes) in 0.0 secs. 060215 151346 java.io.IOException: Impossible situation. There is a score-edit for http://www.legis.state.ak.us/fhelp/helpmain.htm, which comes after the current Page http://www.legis.state.ak.us/folhome.htm 060215 151346 at org.apache.nutch.tools.DistributedAnalysisTool.completeRound( DistributedAnalysisTool.java:532) 060215 151346 at org.apache.nutch.tools.LinkAnalysisTool.iterate( LinkAnalysisTool.java:60) 060215 151346 at org.apache.nutch.tools.LinkAnalysisTool.main( LinkAnalysisTool.java:81)
