Ken Krugler (JIRA) wrote:
[ http://issues.apache.org/jira/browse/NUTCH-353?page=comments#action_12439304 ]
Ken Krugler commented on NUTCH-353:
-----------------------------------
+1 that the redirect target is not always the "real" URL that we want to keep.
For example, http://www.ibm.com/developerworks/lotus/downloads/toolkits.html =>
http://www-128.ibm.com/developerworks/lotus/downloads/toolkits.html. This holds true for most
(all?) developerWorks pages; they redirect to www-128.ibm.com/<whatever>, but IBM would
love for the URL everybody sees to still be www.ibm.com/<whatever>.
If you check status code of the original URL you get 302 Found. By
definition
10.3.3 302 Found
The requested resource resides temporarily under a different URI. Since
the redirection might be altered on occasion, the client SHOULD continue
to use the Request-URI for future requests. This response is only
cacheable if indicated by a Cache-Control or Expires header field.
In this case there is no need to replace original url with redirected.
I know that a lot of sites use permanent redirects in such cases. But I
don't se any proper solution for both.
regards
Uros
pages that serverside forwards will be refetched every time
-----------------------------------------------------------
Key: NUTCH-353
URL: http://issues.apache.org/jira/browse/NUTCH-353
Project: Nutch
Issue Type: Bug
Affects Versions: 0.8.1, 0.9.0
Reporter: Stefan Groschupf
Assigned To: Andrzej Bialecki
Priority: Blocker
Fix For: 0.9.0
Attachments: doNotRefecthForwarderPagesV1.patch
Pages that do a serverside forward are not written with a status change back into the crawlDb. Also the nextFetchTime is not changed.
This causes a refetch of the same page again and again. The result is nutch is not polite and refetching the forwarding and target page in each segment iteration. Also it effects the scoring since the forward page contribute it's score to all outlinks.
