duplicate pages - virtual hosts in db.
--------------------------------------
Key: NUTCH-70
URL: http://issues.apache.org/jira/browse/NUTCH-70
Project: Nutch
Type: Bug
Environment: 0,7 dev
Reporter: Lutischán Ferenc
Dear Developers,
I have a problem with nutch:
- There are many sites duplicates in the webdb and in the segments.
The source of this problem is:
- If the site make 'virtual hosts' (like Apache), e.g. www.origo.hu, origo.hu,
origo.matav.hu, origo.matavnet.hu etc.: the result pages are the same, only the
inlinks are differents.
- The ip address is the same.
- When search, all virtualhosts are in the results.
Google only show one of these virtual hosts, the nutch show all. The result
nutch db is larger, and this case slower, than google.
Have any idea, how to remove these duplicates?
Regards,
Ferenc
--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators:
http://issues.apache.org/jira/secure/Administrators.jspa
-
For more information on JIRA, see:
http://www.atlassian.com/software/jira
-------------------------------------------------------
This SF.Net email is sponsored by the 'Do More With Dual!' webinar happening
July 14 at 8am PDT/11am EDT. We invite you to explore the latest in dual
core and dual graphics technology at this free one hour event hosted by HP,
AMD, and NVIDIA. To register visit http://www.hp.com/go/dualwebinar
_______________________________________________
Nutch-developers mailing list
[email protected]
https://lists.sourceforge.net/lists/listinfo/nutch-developers
