Re: [htdig] more blocks returned

[Evan Tishuk]

> It's hard to say for sure what causes database curruption, but certainly > running out of disk space could do it!  While htdig and htmerge are > working, they may use up more space than the final database size.   As I went looking around the DB and BIN directories I noticed a file called "core" and I guess those were the dump files. Do you think it could be that my swap partition is only 250mb?  Or should this have nothing to do with it?    Also, i am using a start.url file to specify which URLs to dig.  Lets say I have URL#1 URL#2 and URL#3 in that file.  When I dig it's going to spider those sites and record information from them.  Now I merge that info into a database that htdig can read.  Lets say I add URL#4 to the  start.url file.  Or simply just decide to dig URL#4, do I have to also re-dig URL 1, 2, & 3 again before I can merge the DB?  Or can I just  point it at URL#4 and merge what's already existing with the new data from that URL?    Thanks,   -Evan       ----- Original Message ----- From: "Gilles Detillieux" <[EMAIL PROTECTED]> To: "Evan Tishuk" <[EMAIL PROTECTED]> Cc: <[EMAIL PROTECTED]> Sent: Thursday, February 15, 2001 4:11 PM Subject: Re: [htdig] more blocks returned than retrieved (was: no subject) > According to Evan Tishuk: > > I originally asked what the below message meant: > > >> DB2 problem...: /home/httpd/htdig/db/db.docdb: put: more blocks returned > > >> than retrieved > > >> Aborted (core dumped) > > > > Reply was: > > >That's a new complaint, but "DB2 problems" are generally a sign of > > >database corruption. > > > > Do you think it could indicate that during the htmerge process disk > > space ran out? > > Has that ever happened to anyone?  my entire DB is about 1 gig in size, > > and other > > files on the server comprise about another gig, but the hard drive has a > > 10 gig > > capacity!!??  What are some common causes of "database corruption"?  Any > > advice.  How would you all troubleshoot this problem if it happened to > > you?  > > It's hard to say for sure what causes database curruption, but certainly > running out of disk space could do it!  While htdig and htmerge are > working, they may use up more space than the final database size. > The wordlist that htdig generates can be much larger than what htmerge > eventually produces.  In the process, there are a couple copies of this > file hanging around, plus the sort temporaries. > > Another cause could be multiple processes trying to update a database > simultaneously, e.g. by running htmerge before htdig completes its job, > or starting htdig via cron while there's still an earlier htdig or > htmerge process that hadn't finished. > > Finally, some corruption problems can be due to obscure bugs that we just > haven't nailed down yet.  If you can point out something that repeatedly > causes database corruption, we'd like to know.  Otherwise, the only > advice we can give is to try rebuilding your database from scratch. > > -- > Gilles R. Detillieux              E-mail: <[EMAIL PROTECTED]> > Spinal Cord Research Centre       WWW:    http://www.scrc.umanitoba.ca/~grdetil > Dept. Physiology, U. of Manitoba  Phone:  (204)789-3766 > Winnipeg, MB  R3E 3J7  (Canada)   Fax:    (204)789-3930 >