hello, I crawled a local intranet and found duplicate urls. The problem was the crawler treated urls case sensitive collecting duplicate urls. i'm looking for a way to alter urlfilter to remove duplicate urls or set the filter ignore case
for example, the crawler collects the two urls below as separate when the urls point to the same page. www.menu.com/Design/index.jsp www.menu.com/design/index.jsp Does anybody know how to fix this problem? Thank you.
