Gilles Detillieux wrote: > According to Paul Gear: > > I have recently installed the Red Hat packages for htdig-3.2.0-1.b4.0.71 > > and htdig-web-3.2.0-1.b4.0.71 on two Red Hat 7.1 systems. I have both > > configured almost identically, using a fairly "out of the box" > > configuration. I then put symlinks to htdig in /etc/cron.hourly and > > htpurge in /etc/cron.daily. (Obviously, this means that neither is > > invoked in verbose mode.) > > Ouch! Talk about a recipe for disaster! There are no lockouts in htdig > and htmerge, so you have to be careful when running them so that you > don't have one process stomping all over a database that another process > is trying to update. What you're doing will virtually guarantee that > this sort of conflict will occur! If you look at /etc/crontab, you'll > see that the cron.daily jobs are run very shortly after the cron.hourly > jobs, so your htpurge process will start running well before htdig has > any hope of finishing. > > To do this properly, you should write a shell script that runs htdig, > followed by htpurge, and only put this script in cron.hourly if you're > certain it'll always run to completion in well under an hour. If it > takes longer, or even if it's cutting it close, you should either run > it only once daily, or set up an entry in your crontab to run it a few > times a day but with more than adequate times between runs. If you > really want to get fancy, you can set up the shell script to manage a > lock file to prevent multiple processes running the script simultaneously. > > > Over the last few days, one of the servers has been giving output like > > this from the daily htpurge: > > ... > > This goes on for 3.7 Mb of output. The header is repeated several > > times, but mostly it is just the hex numbers from the entry offsets. My > > question is: what causes this? Have i somehow got a debugging build? I > > can't even get this from the second machine if i give it 10 '-v's. > > I'd wager that these errors are the result of database corruption > caused by conflicting writes to the database.
Hi Gilles, Thanks for that info. I must have missed that part of the manual. ;-) I put in some locking to keep htdig & htpurge from stepping on each others' toes, and it seems to have worked. Much appreciated. Paul http://paulgear.webhop.net _______________________________________________ htdig-general mailing list <[EMAIL PROTECTED]> To unsubscribe, send a message to <[EMAIL PROTECTED]> with a subject of unsubscribe FAQ: http://htdig.sourceforge.net/FAQ.html
