Hi everyone, I'm constantly encounter this problem when Nutch comes to database closing stage. The crawler causes my system hung and needs to be restarted. Can anyone help me to figure out this! Here is my log file before hanging:
060210 161454 Finishing update 060210 161959 Processing pagesByURL: Sorted 25968131 instructions in 305.23seconds. 060210 161959 Processing pagesByURL: Sorted 85077.25649510205instructions/second 060210 162403 Processing pagesByURL: Merged to new DB containing 4088028 records in 133.31 seconds 060210 162403 Processing pagesByURL: Merged 30665.576475883277records/second 060210 162428 Processing pagesByMD5: Sorted 3479369 instructions in 25.146seconds. 060210 162428 Processing pagesByMD5: Sorted 138366.6984808717instructions/second 060210 162529 Processing pagesByMD5: Merged to new DB containing 4088028 records in 56.115 seconds 060210 162529 Processing pagesByMD5: Merged 72850.89548249131 records/second 060210 163426 Processing linksByMD5: Sorted 25926879 instructions in 536.227seconds. 060210 163426 Processing linksByMD5: Sorted 48350.56608488569instructions/second 060210 164415 Processing linksByMD5: Merged to new DB containing 50840622 records in 465.897 seconds 060210 164415 Processing linksByMD5: Merged 109124.16692960032records/second 060210 164840 Processing linksByURL: Sorted 22072525 instructions in 264.309seconds. 060210 164840 Processing linksByURL: Sorted 83510.30422724916instructions/second 060210 165830 Processing linksByURL: Merged to new DB containing 50840622 records in 478.304 seconds 060210 165830 Processing linksByURL: Merged 106293.53298320733records/second 060210 170433 Processing linksByMD5: Sorted 23422502 instructions in 362.723seconds. 060210 170433 Processing linksByMD5: Sorted 64574.0744314532instructions/second 060210 171409 Processing linksByMD5: Merged to new DB containing 50840622 records in 453.612 seconds 060210 171409 Processing linksByMD5: Merged 112079.53493293827records/second The weird thing is that Nutch processed linksByMD5 twice, and number of instructions are not the same. Regards, Giang
