[
https://issues.apache.org/jira/browse/NUTCH-1525?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13570005#comment-13570005
]
lufeng commented on NUTCH-1525:
-------------------------------
Hi Lewis,
Yes, The redirection of current URL will be ignored if toHost is equal to
fromHost. Do you mean we would coding like this.
{code:java}
if (ignoreExternalLinks) {
String toHost = new URL(newUrl).getHost().toLowerCase();
String fromHost = new URL(url).getHost().toLowerCase();
if (toHost == null || !toHost.equals(fromHost)) {
// external links
// return;
// indicate that the newUrl is an external link, not ignore
page.putToMetadata(FetcherJob.EXTERNAL_LINKS, TableUtil.YES_VAL);
}
}
{code}
Still Put the redirection url to the Outlinks in current URL even when
db.ignore.external.links set to true. But Maybe in DbUpdate processing, the
redirection url will be updated to WebDb and fetch in next fetching.
> Generator to record external links even when db.ignore.external.links set to
> true
> ----------------------------------------------------------------------------------
>
> Key: NUTCH-1525
> URL: https://issues.apache.org/jira/browse/NUTCH-1525
> Project: Nutch
> Issue Type: Improvement
> Components: generator
> Reporter: Lewis John McGibbney
> Priority: Minor
> Fix For: 1.7, 2.2
>
>
> When fetching pages from specific domains we have various options e.g. use
> urlfilters, set the above property to true before injecting urls into the
> webdb etc. However with the former, it is recognised that complex regex can
> slow down processing and with the latter it means we disregard a number of
> urls which could potentially become useful in the future.
> Unfortunately there is no way to record external links encountered for future
> processing, although the wiki suggests that a very small patch to the
> generator code can allow you to log these links to hadoop.log. although this
> is better, a more robusts storage mechanism would be preferred. This may tie
> in with custom counters we've already specified or may require new counters
> to be implemented.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
