Re: Filtering out duplicate documents...

[Erik Hatcher]

My impression is the new term vector support should at least make this 
type of comparison feasible in some manner.  I'd be interested to see 
what you come up with if you give this a try.  You will need the latest 
CVS codebase.

	Erik

On Mar 8, 2004, at 4:37 PM, Michael Giles wrote:

I'm looking for a way to filter out duplicate documents from an index 
(either while indexing, or after the fact).  It seems like there 
should be an approach of comparing the terms for two documents, but 
I'm wondering if any other folks (i.e. nutch) have come up with a 
solution to this problem.

Obviously you can compute the Levenstein distance on the text, but 
that is way too computationally intensive to scale.  So the goal is to 
find something that would be workable in a production system.  For 
example, a given NYT article, and its printer friendly version should 
be deemed to be the same.

-Mike



---------------------------------------------------------------------
To unsubscribe, e-mail: [EMAIL PROTECTED]
For additional commands, e-mail: [EMAIL PROTECTED]


---------------------------------------------------------------------
To unsubscribe, e-mail: [EMAIL PROTECTED]
For additional commands, e-mail: [EMAIL PROTECTED]