Duplicate URLs with slightly different URIs.. how to normalize?

[Brian Whitman]

I'm using Solr to search the Nutch Lucene index (can't use the nutch  
searcher in our current app.) Using the latest Nutch nightly.

There are a lot of duplicate URLs in the Lucene index-- http:// 
url.com/ vs. http://url.com are two different Lucene documents, as  
are http://url.com/index.html#a and http://url.com/index.html

The Nutch search jsp seems to have some intelligence to remove the  
duplicates -- with the "show all hits" toggle button at the end of  
the results.

Is there a tool to remove duplicates directly from the Lucene index?  
I do call 'nutch dedup' in my crawl script but it doesn't seem to  
affect the results.