Generator and lib-http use different definitions of "unique host"
-----------------------------------------------------------------
Key: NUTCH-268
URL: http://issues.apache.org/jira/browse/NUTCH-268
Project: Nutch
Type: Bug
Versions: 0.8-dev
Reporter: Andrzej Bialecki
Assigned to: Andrzej Bialecki
Fix For: 0.8-dev
Generator uses a host name, as extracted from URL, to determine the maximum
number of URLs from a unique host (when generator.max.per.host is set > 0).
This supposedly should prevent the situation where fetchlists become dominated
by URLs coming from the same hosts, which in turn would clash with "politeness"
rules.
However, http plugins (lib-http HttpBase.blockAddr) don't use host name, and
instead use it's IP address (explicitly doing a DNS lookup on the host name
extracted from URL). This leads to the following undesirable behavior:
* if DNS name resolves to different IPs (round-robin balancing), then
technically we are in violation of the "politeness" rules, because lib-http
doesn't see this as a conflict and permits concurrent accesses to the same host
name.
* if different DNS names resolve to the same IP address (very common: CNAME-s,
subdomains, web hosting, etc) then the purpose of generate.max.per.host is
defeated, because lib-http will block more frequently than intended, leading to
excessive numbers of "Exceeded http.max.delays" exceptions.
Proposed solution: synchronize Generator and lib-http in their interpretation
of "unique host". Introduce a boolean property which instructs both Generator
and lib-http to use in both places either IP addresses or host names as "unique
hosts".
--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators:
http://issues.apache.org/jira/secure/Administrators.jspa
-
For more information on JIRA, see:
http://www.atlassian.com/software/jira
-------------------------------------------------------
Using Tomcat but need to do more? Need to support web services, security?
Get stuff done quickly with pre-integrated technology to make your job easier
Download IBM WebSphere Application Server v.1.0.1 based on Apache Geronimo
http://sel.as-us.falkag.net/sel?cmd=lnk&kid=120709&bid=263057&dat=121642
_______________________________________________
Nutch-developers mailing list
[email protected]
https://lists.sourceforge.net/lists/listinfo/nutch-developers
