Hi, Wahib: The /usr/bin/rundig needs to be executed for taking the new setting in and create the /var/lib/htdig/db.urls when htdig indexing process completes.
If you have already rerun the rundig indexing process, try to locate db.urls. This file should be in the same directory where the indexing databases are. Regards, Jin -----Original Message----- From: Wahib Nackad [mailto:[EMAIL PROTECTED]] Sent: Friday, March 08, 2002 1:07 PM To: [EMAIL PROTECTED]; [EMAIL PROTECTED] Subject: RE: [htdig] Unable to index more than 503 pages Hello Jin, Thanks for the reply. I've added the "create_url_list true" option into the htdig.conf file but without success. No 'db.urls' file is available :(. >Are you sure that all 5,500 documents are located in >http://www.workers.org/ Yes, pretty sure and the site is running on a virtual web server. I'm really lost here. Wahib, >From: "Tsai, Jin" <[EMAIL PROTECTED]> >To: "'Wahib Nackad'" <[EMAIL PROTECTED]>, >[EMAIL PROTECTED] >Subject: RE: [htdig] Unable to index more than 503 pages >Date: Fri, 8 Mar 2002 12:19:41 -0500 > >Hi, Wahib: > >Are you sure that all 5,500 documents are located in >http://www.workers.org/ >web server? Could the documents on the other servers? > >Try turn on the create_url_list to true in your htdig.conf to record the >web >servers htdig has seen. The record result will be in >/var/lib/htdig/db.urls >files in ASCII format. > >Best Regards, > >Jin Tsai >Florida Hospital, MIS > > > >-----Original Message----- >From: Wahib Nackad [mailto:[EMAIL PROTECTED]] >Sent: Friday, March 08, 2002 11:36 AM >To: [EMAIL PROTECTED] >Subject: [htdig] Unable to index more than 503 pages > > >Hi, > >We have a web site with more than 5500 web pages in html format to index >with htdig version 3.2.0b4. Unfortunately, htdig is not able to index more >than 503 pages even if we have 5500 pages. The config is as follow: >start_url: http://www.workers.org/ >limit_urls_to: ${start_url} > >We do not have any robots.txt file blocking anything from the 5500 >available > >pages to index. Many "url rejected: (level 1)" message are returned when we >use "/usr/bin/htdig -vv -i -s -c /home/httpd/html/search/htdig.conf" to >create the db. > >Does someone have an answer to this problem? > >Kind regards, > >_________________________________________________________________ >Get your FREE download of MSN Explorer at http://explorer.msn.com/intl.asp. > > >_______________________________________________ >htdig-general mailing list <[EMAIL PROTECTED]> >To unsubscribe, send a message to ><[EMAIL PROTECTED]> with a subject of unsubscribe >FAQ: http://htdig.sourceforge.net/FAQ.html _________________________________________________________________ Get your FREE download of MSN Explorer at http://explorer.msn.com/intl.asp. _______________________________________________ htdig-general mailing list <[EMAIL PROTECTED]> To unsubscribe, send a message to <[EMAIL PROTECTED]> with a subject of unsubscribe FAQ: http://htdig.sourceforge.net/FAQ.html
