>Date: Tue, 8 Dec 1998 15:31:05 -0600 (CST)
>From: <[EMAIL PROTECTED]>
>
>
>Is/has anyone used htdig for indexing large seach engine data like yahoo
>or altavista? I understand the resources that are required to index such
>a large database, but I would still like to know if someone is/has doing
>it.
>
>db
Recently we started to use htdig remote features for our Virtual Library.
Once we had some wrong Limit_URLs set, so we got some problems.
At the moment we don't take any risk and fetsh 40 MB of 2427 documents
of 165 Server. It took approx. 2 h, but most of the pages have been already
in the proxy server.
Everday we increase the content, we would like to do more, however not
like ALTA VISTA.
We have in mind to do more than 10 000 pages,
I also would like to know how to estimate the limits.
Rolf
------------------------------------------------------------
NDT.net
The e-Journal of Nondestructive Testing & Ultrasonics
Plus NDT online Exhibition
* NDTnet - http://www.ndt.net *
------------------------------------------------------------
NDT Internet Publishing Tel: +49(0)5221-769314
Rolf Diederichs FAX: +49(0)5221-769731
Tacheniusweg 8 Email: [EMAIL PROTECTED]
D-32052 Herford
----------------------------------------------------------------------
To unsubscribe from the htdig mailing list, send a message to
[EMAIL PROTECTED] containing the single word "unsubscribe" in
the body of the message.
