Sorry Andrzej,
I mean on DeleteDuplicates.java, not in runtime. Is that the correct
place to integrate some like Shingling or n-gram?
Massimo
Andrzej Bialecki ha scritto:
Massimo Miccoli wrote:
Hi Doug,
Many thanks for your patch. I now try it. I'm also thinking to
integrate some algo for near duplicated urls detection. I mean some
like Shingling.
Is dedup the best place to integrate the algo?
That would be lovely. Dedup is the place to start, but certainly not
the place to stop... ;-)
I think we should introduce a separate "dedup" field for each page in
the DB. The reason is that if we re-use the md5 (or change its
semantics to mean "near duplicates covered by this value") then we run
a risk of loosing a lot of legitimate unique urls from the DB.
Shingling, if you know how to implement it efficiently, would
certainly be nice - but we could start by just passing a "normalized
text" to md5. By "normalized text" I mean all lowercase, stopwords
removed, punctuation removed, any consecutive whitespace replaced with
exactly 1 space character. We could also use an n-gram profile (either
word-level or character level) with coarse quantization.
