David Odmark wrote:
Hi,
We are using 0.8, and I see a property called db.ignore.internal.links
that is used by LinkDB to, well, ignore internal links. What we need
is a runtime-switchable option that allows the opposite -something
like db.ignore.external links. This is to say, for a given page, we
don't want to follow any inlinks that call out to a different host
from the containing page.
Is there any way to accomplish this without a) modifying Nutch itself,
or b) using an URLFilter and therefore having to maintain a static
file-based list of hosts to exclude?
Thanks in advance,
David Odmark
+1
That would be really useful in one of the projects I am using nutch for.
It would be a mess to implement I think... Humm... maybe not.
LinkDb.java
line 44 insert
private boolean ignoreExternalLinks;
line 55 insert
ignoreExternalLinks = job.getBoolean("db.ignore.external.links",false);
line 82
insert something like
if(ignoreExternalLinks) {
String toHost = getHost(toUrl);
if (toHost == null || !toHost.equals(fromHost)) { // external link
continue; // skip it
}
}
Would that code be almost correct? David you want to try that?
Jeff
-------------------------------------------------------
This SF.Net email is sponsored by xPML, a groundbreaking scripting language
that extends applications into web and mobile media. Attend the live webcast
and join the prime developer group breaking into this new coding territory!
http://sel.as-us.falkag.net/sel?cmd=lnk&kid=110944&bid=241720&dat=121642
_______________________________________________
Nutch-general mailing list
[email protected]
https://lists.sourceforge.net/lists/listinfo/nutch-general
