logging at info gives a rather long stack...  here it is.
the funny thing is, after the crash, i deleted the tmp-dir in the db folder
(without cleaning up the segment folders), and recrawl the same site, and it
worked without crashing.

060215 151338 Using URL normalizer: org.apache.nutch.net.BasicUrlNormalizer
060215 151338 Added 1 pages
060215 151338 Processing pagesByURL: Sorted 1 instructions in 0.01 seconds.
060215 151338 Processing pagesByURL: Sorted 100.0 instructions/second
060215 151338 Processing pagesByURL: Merged to new DB containing 1 records
in 0.01 seconds
060215 151338 Processing pagesByURL: Merged 100.0 records/second
060215 151338 Processing pagesByMD5: Sorted 1 instructions in 0.0 seconds.
060215 151338 Processing pagesByMD5: Sorted Infinity instructions/second
060215 151338 Processing pagesByMD5: Merged to new DB containing 1 records
in 0.0 seconds
060215 151338 Processing pagesByMD5: Merged Infinity records/second
060215 151338 Processing linksByMD5: Copied file (4096 bytes) in 0.0 secs.
060215 151338 Processing linksByURL: Copied file (4096 bytes) in 0.0 secs.
060215 151338 Processing pagesByURL: Sorted 1 instructions in 0.0 seconds.
060215 151338 Processing pagesByURL: Sorted Infinity instructions/second
060215 151338 Processing pagesByURL: Merged to new DB containing 1 records
in 0.0 seconds
060215 151338 Processing pagesByURL: Merged Infinity records/second
060215 151338 Processing pagesByMD5: Sorted 1 instructions in 0.0 seconds.
060215 151338 Processing pagesByMD5: Sorted Infinity instructions/second
060215 151338 Processing pagesByMD5: Merged to new DB containing 1 records
in 0.0 seconds
060215 151338 Processing pagesByMD5: Merged Infinity records/second
060215 151338 Processing linksByMD5: Copied file (4096 bytes) in 0.0 secs.
060215 151338 Processing linksByURL: Copied file (4096 bytes) in 0.0 secs.
060215 151338 Processing
/var/nutch/indexes-ak/segments/20060215151338/fetchlist.unsorted: Sorted 1
entries in 0.0 seconds.
060215 151338 Processing
/var/nutch/indexes-ak/segments/20060215151338/fetchlist.unsorted: Sorted
Infinity entries/second
060215 151338 Overall processing: Sorted 1 entries in 0.0 seconds.
060215 151338 Overall processing: Sorted 0.0 entries/second
060215 151339 logging at INFO
060215 151339 fetching http://www.legis.state.ak.us/default.htm
060215 151339 http.proxy.host = null
060215 151339 http.proxy.port = 8080
060215 151339 http.timeout = 10000
060215 151339 http.content.limit = -1
060215 151339 http.agent = NutchCVS/0.7.1 (Nutch;
http://lucene.apache.org/nutch/bot.html; [email protected])
060215 151339 fetcher.server.delay = 1000
060215 151339 http.max.delays = 25
060215 151340 status: segment 20060215151338, 1 pages, 0 errors, 8287 bytes,
1020 ms
060215 151340 status: 0.98039216 pages/s, 63.472733 kb/s, 8287.0 bytes/page
060215 151341 Updating /var/nutch/indexes-ak/db
060215 151341 Updating for /var/nutch/indexes-ak/segments/20060215151338
060215 151341 Processing document 0
060215 151341 Finishing update
060215 151341 Processing pagesByURL: Sorted 6 instructions in 0.0 seconds.
060215 151341 Processing pagesByURL: Sorted Infinity instructions/second
060215 151341 Processing pagesByURL: Merged to new DB containing 6 records
in 0.0 seconds
060215 151341 Processing pagesByURL: Merged Infinity records/second
060215 151341 Processing pagesByMD5: Sorted 7 instructions in 0.0 seconds.
060215 151341 Processing pagesByMD5: Sorted Infinity instructions/second
060215 151341 Processing pagesByMD5: Merged to new DB containing 6 records
in 0.0 seconds
060215 151341 Processing pagesByMD5: Merged Infinity records/second
060215 151341 Processing linksByMD5: Sorted 6 instructions in 0.0 seconds.
060215 151341 Processing linksByMD5: Sorted Infinity instructions/second
060215 151341 Processing linksByMD5: Merged to new DB containing 5 records
in 0.01 seconds
060215 151341 Processing linksByMD5: Merged 500.0 records/second
060215 151341 Processing linksByURL: Sorted 5 instructions in 0.0 seconds.
060215 151341 Processing linksByURL: Sorted Infinity instructions/second
060215 151341 Processing linksByURL: Merged to new DB containing 5 records
in 0.0 seconds
060215 151341 Processing linksByURL: Merged Infinity records/second
060215 151341 Processing linksByMD5: Sorted 5 instructions in 0.0 seconds.
060215 151341 Processing linksByMD5: Sorted Infinity instructions/second
060215 151341 Processing linksByMD5: Merged to new DB containing 5 records
in 0.0 seconds
060215 151341 Processing linksByMD5: Merged Infinity records/second
060215 151341 Update finished
060215 151341 Processing pagesByURL: Sorted 5 instructions in 0.0 seconds.
060215 151341 Processing pagesByURL: Sorted Infinity instructions/second
060215 151341 Processing pagesByURL: Merged to new DB containing 6 records
in 0.01 seconds
060215 151341 Processing pagesByURL: Merged 600.0 records/second
060215 151341 Processing pagesByMD5: Sorted 5 instructions in 0.0 seconds.
060215 151341 Processing pagesByMD5: Sorted Infinity instructions/second
060215 151341 Processing pagesByMD5: Merged to new DB containing 6 records
in 0.0 seconds
060215 151341 Processing pagesByMD5: Merged Infinity records/second
060215 151341 Processing linksByMD5: Copied file (4096 bytes) in 0.0 secs.
060215 151341 Processing linksByURL: Copied file (4096 bytes) in 0.0 secs.
060215 151341 Processing
/var/nutch/indexes-ak/segments/20060215151341/fetchlist.unsorted: Sorted 5
entries in 0.0 seconds.
060215 151341 Processing
/var/nutch/indexes-ak/segments/20060215151341/fetchlist.unsorted: Sorted
Infinity entries/second
060215 151341 Overall processing: Sorted 5 entries in 0.0 seconds.
060215 151341 Overall processing: Sorted 0.0 entries/second
060215 151341 logging at INFO
060215 151341 fetching http://www.legis.state.ak.us/infolist.htm
060215 151341 fetching http://www.legis.state.ak.us/home.htm
060215 151341 fetching http://www.legis.state.ak.us/fhelp/helpmain.htm
060215 151341 fetching http://www.legis.state.ak.us/FBframe.htm
060215 151341 fetching http://www.legis.state.ak.us/Legframe.htm
060215 151341 fetch of
http://www.legis.state.ak.us/fhelp/helpmain.htmfailed with:
java.lang.Exception: org.apache.nutch.protocol.R
esourceGone: Blocked by robots.txt
060215 151342 status: segment 20060215151341, 4 pages, 1 errors, 16047
bytes, 1010 ms
060215 151342 status: 3.960396 pages/s, 124.12593 kb/s, 4011.75 bytes/page
060215 151343 Updating /var/nutch/indexes-ak/db
060215 151343 Updating for /var/nutch/indexes-ak/segments/20060215151341
060215 151343 Processing document 0
060215 151343 Finishing update
060215 151343 Processing pagesByURL: Sorted 10 instructions in 0.0 seconds.
060215 151343 Processing pagesByURL: Sorted Infinity instructions/second
060215 151343 Processing pagesByURL: Merged to new DB containing 9 records
in 0.0 seconds
060215 151343 Processing pagesByURL: Merged Infinity records/second
060215 151343 Processing pagesByMD5: Sorted 13 instructions in 0.0 seconds.
060215 151343 Processing pagesByMD5: Sorted Infinity instructions/second
060215 151343 Processing pagesByMD5: Merged to new DB containing 9 records
in 0.0 seconds
060215 151343 Processing pagesByMD5: Merged Infinity records/second
060215 151343 Processing linksByMD5: Sorted 11 instructions in 0.0 seconds.
060215 151343 Processing linksByMD5: Sorted Infinity instructions/second
060215 151343 Processing linksByMD5: Merged to new DB containing 11 records
in 0.0 seconds
060215 151343 Processing linksByMD5: Merged Infinity records/second
060215 151343 Processing linksByURL: Sorted 6 instructions in 0.0 seconds.
060215 151343 Processing linksByURL: Sorted Infinity instructions/second
060215 151343 Processing linksByURL: Merged to new DB containing 9 records
in 0.01 seconds
060215 151343 Processing linksByURL: Merged 900.0 records/second
060215 151343 Processing linksByMD5: Sorted 9 instructions in 0.0 seconds.
060215 151343 Processing linksByMD5: Sorted Infinity instructions/second
060215 151343 Processing linksByMD5: Merged to new DB containing 9 records
in 0.0 seconds
060215 151343 Processing linksByMD5: Merged Infinity records/second
060215 151343 Update finished
060215 151343 Processing pagesByURL: Sorted 4 instructions in 0.0 seconds.
060215 151343 Processing pagesByURL: Sorted Infinity instructions/second
060215 151343 Processing pagesByURL: Merged to new DB containing 9 records
in 0.0 seconds
060215 151343 Processing pagesByURL: Merged Infinity records/second
060215 151343 Processing pagesByMD5: Sorted 4 instructions in 0.0 seconds.
060215 151343 Processing pagesByMD5: Sorted Infinity instructions/second
060215 151343 Processing pagesByMD5: Merged to new DB containing 9 records
in 0.0 seconds
060215 151343 Processing pagesByMD5: Merged Infinity records/second
060215 151343 Processing linksByMD5: Copied file (4096 bytes) in 0.0 secs.
060215 151343 Processing linksByURL: Copied file (4096 bytes) in 0.0 secs.
060215 151343 Processing
/var/nutch/indexes-ak/segments/20060215151343/fetchlist.unsorted: Sorted 4
entries in 0.0 seconds.
060215 151343 Processing
/var/nutch/indexes-ak/segments/20060215151343/fetchlist.unsorted: Sorted
Infinity entries/second
060215 151343 Overall processing: Sorted 4 entries in 0.0 seconds.
060215 151343 Overall processing: Sorted 0.0 entries/second
060215 151343 logging at INFO
060215 151343 fetching http://www.legis.state.ak.us/
060215 151343 fetching http://www.legis.state.ak.us/folhome.htm
060215 151343 fetching http://www.legis.state.ak.us/poms/
060215 151343 fetch of http://www.legis.state.ak.us/poms/ failed with:
java.lang.Exception: org.apache.nutch.protocol.ResourceGone:
Blocked by robots.txt
060215 151343 fetching http://www.legis.state.ak.us/basis/start.asp
060215 151343 fetch okay, but can't parse
http://www.legis.state.ak.us/basis/start.asp, reason: failed(2,200):
sun.io.MalformedInput
Exception: Missing byte-order mark
060215 151344 status: segment 20060215151343, 3 pages, 1 errors, 23109
bytes, 1010 ms
060215 151344 status: 2.970297 pages/s, 178.75154 kb/s, 7703.0 bytes/page
060215 151345 Updating /var/nutch/indexes-ak/db
060215 151345 Updating for /var/nutch/indexes-ak/segments/20060215151343
060215 151345 Processing document 0
060215 151345 Finishing update
060215 151345 Processing pagesByURL: Sorted 5 instructions in 0.0 seconds.
060215 151345 Processing pagesByURL: Sorted Infinity instructions/second
060215 151345 Processing pagesByURL: Merged to new DB containing 9 records
in 0.0 seconds
060215 151345 Processing pagesByURL: Merged Infinity records/second
060215 151345 Processing pagesByMD5: Sorted 8 instructions in 0.0 seconds.
060215 151345 Processing pagesByMD5: Sorted Infinity instructions/second
060215 151345 Processing pagesByMD5: Merged to new DB containing 9 records
in 0.0 seconds
060215 151345 Processing pagesByMD5: Merged Infinity records/second
060215 151345 Processing linksByMD5: Sorted 13 instructions in 0.0 seconds.
060215 151345 Processing linksByMD5: Sorted Infinity instructions/second
060215 151345 Processing linksByMD5: Merged to new DB containing 14 records
in 0.0 seconds
060215 151345 Processing linksByMD5: Merged Infinity records/second
060215 151345 Processing linksByURL: Sorted 9 instructions in 0.0 seconds.
060215 151345 Processing linksByURL: Sorted Infinity instructions/second
060215 151345 Processing linksByURL: Merged to new DB containing 14 records
in 0.01 seconds
060215 151345 Processing linksByURL: Merged 1400.0 records/second
060215 151345 Processing linksByMD5: Sorted 10 instructions in 0.0 seconds.
060215 151345 Processing linksByMD5: Sorted Infinity instructions/second
060215 151345 Processing linksByMD5: Merged to new DB containing 14 records
in 0.0 seconds
060215 151345 Processing linksByMD5: Merged Infinity records/second
060215 151345 Update finished
060215 151345 Processing pagesByURL: Sorted 1 instructions in 0.0 seconds.
060215 151345 Processing pagesByURL: Sorted Infinity instructions/second
060215 151345 Processing pagesByURL: Merged to new DB containing 9 records
in 0.0 seconds
060215 151345 Processing pagesByURL: Merged Infinity records/second
060215 151345 Processing pagesByMD5: Sorted 1 instructions in 0.0 seconds.
060215 151345 Processing pagesByMD5: Sorted Infinity instructions/second
060215 151345 Processing pagesByMD5: Merged to new DB containing 9 records
in 0.0 seconds
060215 151345 Processing pagesByMD5: Merged Infinity records/second
060215 151345 Processing linksByMD5: Copied file (4096 bytes) in 0.0 secs.
060215 151345 Processing linksByURL: Copied file (4096 bytes) in 0.0 secs.
060215 151345 Processing
/var/nutch/indexes-ak/segments/20060215151345/fetchlist.unsorted: Sorted 1
entries in 0.0 seconds.
060215 151345 Processing
/var/nutch/indexes-ak/segments/20060215151345/fetchlist.unsorted: Sorted
Infinity entries/second
060215 151345 Overall processing: Sorted 1 entries in 0.0 seconds.
060215 151345 Overall processing: Sorted 0.0 entries/second
060215 151345 logging at INFO
060215 151345 fetching http://www.legis.state.ak.us/fhelp/helpmain.htm
060215 151345 fetch of
http://www.legis.state.ak.us/fhelp/helpmain.htmfailed with:
java.lang.Exception: org.apache.nutch.protocol.R
esourceGone: Blocked by robots.txt
060215 151346 Updating /var/nutch/indexes-ak/db
060215 151346 Updating for /var/nutch/indexes-ak/segments/20060215151345
060215 151346 Processing document 0
060215 151346 Finishing update
060215 151346 Processing pagesByURL: Sorted 1 instructions in 0.0 seconds.
060215 151346 Processing pagesByURL: Sorted Infinity instructions/second
060215 151346 Processing pagesByURL: Merged to new DB containing 8 records
in 0.0 seconds
060215 151346 Processing pagesByURL: Merged Infinity records/second
060215 151346 Processing pagesByMD5: Sorted 1 instructions in 0.0 seconds.
060215 151346 Processing pagesByMD5: Sorted Infinity instructions/second
060215 151346 Processing pagesByMD5: Merged to new DB containing 8 records
in 0.0 seconds
060215 151346 Processing pagesByMD5: Merged Infinity records/second
060215 151346 Processing linksByMD5: Sorted 1 instructions in 0.01 seconds.
060215 151346 Processing linksByMD5: Sorted 100.0 instructions/second
060215 151346 Processing linksByMD5: Merged to new DB containing 14 records
in 0.0 seconds
060215 151346 Processing linksByMD5: Merged Infinity records/second
060215 151346 Processing linksByURL: Copied file (4096 bytes) in 0.0 secs.
060215 151346 Update finished
060215 151346 Overall processing: Sorted 0 entries in 0.0 seconds.
060215 151346 Overall processing: Sorted NaN entries/second
060215 151346 Deleted empty fetchlist segment dir
/var/nutch/indexes-ak/segments/20060215151346
060215 151346 Start at: 0
060215 151346 Extent: 8
060215 151346 Pages consumed: 0 (at index 0). Links fetched: 0.
060215 151346 Opened stream to file 0
060215 151346 Pages written: 0
060215 151346 Processing pagesByURL: Sorted 5 instructions in 0.0 seconds.
060215 151346 Processing pagesByURL: Sorted Infinity instructions/second
060215 151346 Processing pagesByURL: Merged to new DB containing 8 records
in 0.0 seconds
060215 151346 Processing pagesByURL: Merged Infinity records/second
060215 151346 Processing pagesByMD5: Sorted 5 instructions in 0.01 seconds.
060215 151346 Processing pagesByMD5: Sorted 500.0 instructions/second
060215 151346 Processing pagesByMD5: Merged to new DB containing 8 records
in 0.0 seconds
060215 151346 Processing pagesByMD5: Merged Infinity records/second
060215 151346 Processing linksByMD5: Copied file (4096 bytes) in 0.0 secs.
060215 151346 Processing linksByURL: Copied file (4096 bytes) in 0.0 secs.
060215 151346 java.io.IOException: Impossible situation.  There is a
score-edit for http://www.legis.state.ak.us/fhelp/helpmain.htm,
 which comes after the current Page http://www.legis.state.ak.us/folhome.htm
060215 151346 at
org.apache.nutch.tools.DistributedAnalysisTool.completeRound(
DistributedAnalysisTool.java:532)
060215 151346 at org.apache.nutch.tools.LinkAnalysisTool.iterate(
LinkAnalysisTool.java:60)
060215 151346 at org.apache.nutch.tools.LinkAnalysisTool.main(
LinkAnalysisTool.java:81)

Reply via email to