Hi, We have a web site with more than 5500 web pages in html format to index with htdig version 3.2.0b4. Unfortunately, htdig is not able to index more than 503 pages even if we have 5500 pages. The config is as follow: start_url: http://www.workers.org/ limit_urls_to: ${start_url}
We do not have any robots.txt file blocking anything from the 5500 available pages to index. Many "url rejected: (level 1)" message are returned when we use "/usr/bin/htdig -vv -i -s -c /home/httpd/html/search/htdig.conf" to create the db. Does someone have an answer to this problem? Kind regards, _________________________________________________________________ Get your FREE download of MSN Explorer at http://explorer.msn.com/intl.asp. _______________________________________________ htdig-general mailing list <[EMAIL PROTECTED]> To unsubscribe, send a message to <[EMAIL PROTECTED]> with a subject of unsubscribe FAQ: http://htdig.sourceforge.net/FAQ.html
