[ http://issues.apache.org/jira/browse/NUTCH-13?page=comments#action_63416 ] Matthias Jaekle commented on NUTCH-13: --------------------------------------
If fetcher is the only task which already runs the dns-lookup it might be the best place to implement the ip filter there to avoid making dns lookups multiple times. If the IP-filter blocks an ip, we maybe simply could handle the page as pages which return 403 (Forbidden). > If dns points to 127.0.0.1, the url is also crawled > --------------------------------------------------- > > Key: NUTCH-13 > URL: http://issues.apache.org/jira/browse/NUTCH-13 > Project: Nutch > Type: Bug > Components: fetcher > Reporter: Matthias Jaekle > Priority: Minor > > For example www.tik24.de points to 127.0.0.1. > If you follow a link to www.tik24.de fetcher will crawl content from your own > machine. > Wrong DNS entries could create unwanted entries in segments. -- This message is automatically generated by JIRA. - If you think it was sent incorrectly contact one of the administrators: http://issues.apache.org/jira/secure/Administrators.jspa - For more information on JIRA, see: http://www.atlassian.com/software/jira ------------------------------------------------------- SF email is sponsored by - The IT Product Guide Read honest & candid reviews on hundreds of IT Products from real users. Discover which products truly live up to the hype. Start reading now. http://ads.osdn.com/?ad_id=6595&alloc_id=14396&op=click _______________________________________________ Nutch-developers mailing list [email protected] https://lists.sourceforge.net/lists/listinfo/nutch-developers
