The site I am indexing is a bit peculiar. The following
is an example of the setup, where each page is exactly
the same.
www.domain.com/subdirectory/
www.domain.com/subdirectory/index.html
www.domain.com/Subdirectory/
www.domain.com/Subdirectory/index.html
I assumed that in the case where there is no index.html
that it was just loading the index.html. Here's the
problem. htdig recognizes this as 4 different pages,
and indexes all of them. I can see where it would think
it is 2 different because of the s and S. Is there any
way to prevent the duplicates?
Thanks!
Adam
------------------------------------
To unsubscribe from the htdig mailing list, send a message to
[EMAIL PROTECTED]
You will receive a message to confirm this.
List archives: <http://www.htdig.org/mail/menu.html>
FAQ: <http://www.htdig.org/FAQ.html>
