Hi, Wahib: Are you sure that all 5,500 documents are located in http://www.workers.org/ web server? Could the documents on the other servers?
Try turn on the create_url_list to true in your htdig.conf to record the web servers htdig has seen. The record result will be in /var/lib/htdig/db.urls files in ASCII format. Best Regards, Jin Tsai Florida Hospital, MIS -----Original Message----- From: Wahib Nackad [mailto:[EMAIL PROTECTED]] Sent: Friday, March 08, 2002 11:36 AM To: [EMAIL PROTECTED] Subject: [htdig] Unable to index more than 503 pages Hi, We have a web site with more than 5500 web pages in html format to index with htdig version 3.2.0b4. Unfortunately, htdig is not able to index more than 503 pages even if we have 5500 pages. The config is as follow: start_url: http://www.workers.org/ limit_urls_to: ${start_url} We do not have any robots.txt file blocking anything from the 5500 available pages to index. Many "url rejected: (level 1)" message are returned when we use "/usr/bin/htdig -vv -i -s -c /home/httpd/html/search/htdig.conf" to create the db. Does someone have an answer to this problem? Kind regards, _________________________________________________________________ Get your FREE download of MSN Explorer at http://explorer.msn.com/intl.asp. _______________________________________________ htdig-general mailing list <[EMAIL PROTECTED]> To unsubscribe, send a message to <[EMAIL PROTECTED]> with a subject of unsubscribe FAQ: http://htdig.sourceforge.net/FAQ.html _______________________________________________ htdig-general mailing list <[EMAIL PROTECTED]> To unsubscribe, send a message to <[EMAIL PROTECTED]> with a subject of unsubscribe FAQ: http://htdig.sourceforge.net/FAQ.html
