Hello,
How are people dealing with avoiding page duplication where URL are similar,
but the content is identical. I know there is page content fingerprinting and
shingling (MD5Signature and TextProfileSignature), but that assumes you
already fetched the content. I am wondering if its possible to detect things
earlier than that, even if it's not 10)% reliable.
Concretely, imagine the following URLs:
http://example.com
http://www.example.com
http://www1.example.com
http://www2.example.com
They are all, very likely, pointing to the same page. One person may link to
www.example.com and the other person may link to just example.com, thus
we parse 2 different URLs, when ideally we'd want just a single URL for each
page. Similarly, the example site may have multiple web servers (e.g. for
load balancing), but each with a slightly different name (e.g. www1.... ,
www2....),
pointing to the same site.
What's the best way to treat www1 and www2 as just www?
Are people using regex-normalize.xml for that?
Thanks,
Otis
--
Sematext -- http://sematext.com/ -- Lucene - Solr - Nutch
