According to Evan Tishuk:
> > It's hard to say for sure what causes database curruption, but certainly
> > running out of disk space could do it! While htdig and htmerge are
> > working, they may use up more space than the final database size.
>
>
> As I went looking around the DB and BIN directories I noticed a file
> called "core" and I guess those were the dump files.
> Do you think it could be that my swap partition is only 250mb?
> Or should this have nothing to do with it?
>
> Also, i am using a start.url file to specify which URLs to dig. Lets
> say I have URL#1 URL#2 and URL#3 in that file. When I dig it's going to
>
> spider those sites and record information from them. Now I merge that
> info into a database that htdig can read. Lets say I add URL#4 to the
> start.url file. Or simply just decide to dig URL#4, do I have to also
> re-dig URL 1, 2, & 3 again before I can merge the DB? Or can I just
> point it at URL#4 and merge what's already existing with the new data
> from that URL?
To create a 1 GB database, it may well be that htdig and/or htmerge need
huge amounts of virtual memory. Depending on how much RAM you have, 250
MB of swap may not be sufficient. It would be worth trying to add swap
or RAM or both. If htdig or htmerge runs out of virtual memory, it would
crash and could leave the databases in an unusable state.
You could find out where the core file came from, using either the "file"
command or the "strings" command, and then get a backtrace from that
program. E.g.:
# file core
core: ELF 32-bit LSB core file of 'htdig' (signal 11), Intel 80386, version 1
# gdb /opt/www/htdig/bin/htdig core
GDB is free...
....
(gdb) bt
....
(gdb) quit
#
That may help get to the root of the problem.
--
Gilles R. Detillieux E-mail: <[EMAIL PROTECTED]>
Spinal Cord Research Centre WWW: http://www.scrc.umanitoba.ca/~grdetil
Dept. Physiology, U. of Manitoba Phone: (204)789-3766
Winnipeg, MB R3E 3J7 (Canada) Fax: (204)789-3930
_______________________________________________
htdig-general mailing list <[EMAIL PROTECTED]>
Information: http://lists.sourceforge.net/lists/listinfo/htdig-general
FAQ: http://htdig.sourceforge.net/FAQ.html
