Hello Jim, Thanks for your reply and help on this issue. You're right, the problem was related to missing links into html pages not pointing to subdirectories on the site. I've added www.workers.org/ww/ and any other subdir and we're able to index more than 5000 pages now. Thanks again for your continue help here.
Wahib >From: Jim Cole <[EMAIL PROTECTED]> >To: Wahib Nackad <[EMAIL PROTECTED]> >CC: <[EMAIL PROTECTED]> >Subject: Re: [htdig] Unable to index more than 503 pages >Date: Fri, 8 Mar 2002 21:07:26 -0700 (MST) > >Wahib Nackad's bits of Sat, 9 Mar 2002 translated to: > > >Yes, Im really tired. Can someone try to index the www.workers.org and >let > >me know how much result of indexed page he receive. This will surelly >help > >me to know if the problem comes from the web server or htdig. > > >... > >>more > >> >than 503 pages even if we have 5500 pages. The config is as follow: > >> >start_url: http://www.workers.org/ > >> >limit_urls_to: ${start_url} > >Do you have any links from the main pages to your archives (e.g. >/ww/2000/ /ww/2001/)? ht://Dig locates documents by following >links, so unless there is some way to traverse links from >www.workers.org/ to the files in these directories, the files >will not be indexed. I suspect that this is your problem. You >might want try adding some of the other directories to your >start_url and see if that makes a difference. For example > >start_url: http://www.workers.org/ http://www.workers.org/ww/2000/ > >Jim > > >_______________________________________________ >htdig-general mailing list <[EMAIL PROTECTED]> >To unsubscribe, send a message to ><[EMAIL PROTECTED]> with a subject of unsubscribe >FAQ: http://htdig.sourceforge.net/FAQ.html _________________________________________________________________ Chat with friends online, try MSN Messenger: http://messenger.msn.com _______________________________________________ htdig-general mailing list <[EMAIL PROTECTED]> To unsubscribe, send a message to <[EMAIL PROTECTED]> with a subject of unsubscribe FAQ: http://htdig.sourceforge.net/FAQ.html
