[ http://issues.apache.org/jira/browse/NUTCH-13?page=comments#action_63418 ] byron miller commented on NUTCH-13: -----------------------------------
If we wan't to support IP's lets do it both ways. Banned list: ipdeny.txt or something similar that contains an ip address range/subnet that will be ignored. 127.0.0.1 192.255.255.255 10.255.255.255 or 192.*.*.* ...... fetch based on ips: ./bin/nutch -generate segments/ -iprange http://10.1.1.[1-255] Create a fetch list for all ip's in that range (or use 10.1.1.255/24 for subnet based filtering..) For intranet searches some places may like not having the overhead of name resolution. Problems with keeping IP's in a public search engine. 1. IP is usually a shared servers main address and has no content. 2. Continuing #1, the ip wouldn't have the http headers that return the sites that could be associated on that IP 3. If either 1 or 2 are true a default site can be on that ip address as well as a FQDN and dupe detection would have to full consider unique hosts with similar content and be smart enough to keep the one with higher priority (possibly a configruation preference) make sense? > If dns points to 127.0.0.1, the url is also crawled > --------------------------------------------------- > > Key: NUTCH-13 > URL: http://issues.apache.org/jira/browse/NUTCH-13 > Project: Nutch > Type: Bug > Components: fetcher > Reporter: Matthias Jaekle > Priority: Minor > > For example www.tik24.de points to 127.0.0.1. > If you follow a link to www.tik24.de fetcher will crawl content from your own > machine. > Wrong DNS entries could create unwanted entries in segments. -- This message is automatically generated by JIRA. - If you think it was sent incorrectly contact one of the administrators: http://issues.apache.org/jira/secure/Administrators.jspa - For more information on JIRA, see: http://www.atlassian.com/software/jira ------------------------------------------------------- SF email is sponsored by - The IT Product Guide Read honest & candid reviews on hundreds of IT Products from real users. Discover which products truly live up to the hype. Start reading now. http://ads.osdn.com/?ad_id=6595&alloc_id=14396&op=click _______________________________________________ Nutch-developers mailing list [email protected] https://lists.sourceforge.net/lists/listinfo/nutch-developers
