Hello Jin, Thanks for the reply. I've added the "create_url_list true" option into the htdig.conf file but without success. No 'db.urls' file is available :(.
>Are you sure that all 5,500 documents are located in >http://www.workers.org/ Yes, pretty sure and the site is running on a virtual web server. I'm really lost here. Wahib, >From: "Tsai, Jin" <[EMAIL PROTECTED]> >To: "'Wahib Nackad'" <[EMAIL PROTECTED]>, >[EMAIL PROTECTED] >Subject: RE: [htdig] Unable to index more than 503 pages >Date: Fri, 8 Mar 2002 12:19:41 -0500 > >Hi, Wahib: > >Are you sure that all 5,500 documents are located in >http://www.workers.org/ >web server? Could the documents on the other servers? > >Try turn on the create_url_list to true in your htdig.conf to record the >web >servers htdig has seen. The record result will be in >/var/lib/htdig/db.urls >files in ASCII format. > >Best Regards, > >Jin Tsai >Florida Hospital, MIS > > > >-----Original Message----- >From: Wahib Nackad [mailto:[EMAIL PROTECTED]] >Sent: Friday, March 08, 2002 11:36 AM >To: [EMAIL PROTECTED] >Subject: [htdig] Unable to index more than 503 pages > > >Hi, > >We have a web site with more than 5500 web pages in html format to index >with htdig version 3.2.0b4. Unfortunately, htdig is not able to index more >than 503 pages even if we have 5500 pages. The config is as follow: >start_url: http://www.workers.org/ >limit_urls_to: ${start_url} > >We do not have any robots.txt file blocking anything from the 5500 >available > >pages to index. Many "url rejected: (level 1)" message are returned when we >use "/usr/bin/htdig -vv -i -s -c /home/httpd/html/search/htdig.conf" to >create the db. > >Does someone have an answer to this problem? > >Kind regards, > >_________________________________________________________________ >Get your FREE download of MSN Explorer at http://explorer.msn.com/intl.asp. > > >_______________________________________________ >htdig-general mailing list <[EMAIL PROTECTED]> >To unsubscribe, send a message to ><[EMAIL PROTECTED]> with a subject of unsubscribe >FAQ: http://htdig.sourceforge.net/FAQ.html _________________________________________________________________ Get your FREE download of MSN Explorer at http://explorer.msn.com/intl.asp. _______________________________________________ htdig-general mailing list <[EMAIL PROTECTED]> To unsubscribe, send a message to <[EMAIL PROTECTED]> with a subject of unsubscribe FAQ: http://htdig.sourceforge.net/FAQ.html
