Ht://Dig uses up nearly a gig of my bandwidth every month. It tends to lap itself if I run it daily, so I've started running it every other day. Otherwise it runs great.
I am now presented with the task of needing to mirror sites, in addition to index them with my search engine. I shudder to think of the resources this will use; both on my web servers to be mirrored/indexed and my bandwidth. Disk space, is not a big concern to me though. Is it possible to create a mirror of a site using the information in ht://dig's databases so that I can save the extra effort of mirroring? I was using rsync to keep my bandwidth low, but now I need to switch to something that works like wget so that I can get static html snapshots instead of the actual cgi/php/asp source pages. Thanks for any info, Matthew Nuzum www.bearfruit.org [EMAIL PROTECTED] _______________________________________________________________ Have big pipes? SourceForge.net is looking for download mirrors. We supply the hardware. You get the recognition. Email Us: [EMAIL PROTECTED] _______________________________________________ htdig-general mailing list <[EMAIL PROTECTED]> To unsubscribe, send a message to <[EMAIL PROTECTED]> with a subject of unsubscribe FAQ: http://htdig.sourceforge.net/FAQ.html
