[
https://issues.apache.org/jira/browse/NUTCH-2377?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15994558#comment-15994558
]
Sebastian Nagel commented on NUTCH-2377:
----------------------------------------
Hi [~abhakim1980],
everything looks fine when trying with parsechecker and the recent head of
Nutch 2.x:
{noformat}
% $NUTCH_HOME/bin/nutch parsechecker https://www.ouedkniss.com
fetching: https://www.ouedkniss.com
parsing: https://www.ouedkniss.com
contentType: text/html
...
outlink: toUrl: https://www.ouedkniss.com/?lang=fr anchor:
outlink: toUrl: https://www.ouedkniss.com/scripts/jquery-1.11.3.min.js
anchor:
outlink: toUrl: https://www.ouedkniss.com/scripts/basic.js?v=286 anchor:
outlink: toUrl: https://www.ouedkniss.com/images/favicon.ico anchor:
...
outlink: toUrl:
https://www.ouedkniss.com/loisirs_divertissements/jeu-video/gameboy anchor:
GameBoy
outlink: toUrl:
https://www.ouedkniss.com/loisirs_divertissements/chasse-et-peche anchor:
Chasse & Peche
{noformat}
447 outlinks, most of them from properly resolved from relative links in the
HTML. After increasing the content limit (see property {{http.content.limit}})
the number of found outlinks is 978:
{noformat}
% nutch parsechecker -Dhttp.content.limit=-1 https://www.ouedkniss.com
...
{noformat}
Please, resolve this issue as "Not A Problem" in case this matches your
expectations. Thanks!
> Nutch can't parse relative links
> ---------------------------------
>
> Key: NUTCH-2377
> URL: https://issues.apache.org/jira/browse/NUTCH-2377
> Project: Nutch
> Issue Type: Bug
> Components: parser
> Affects Versions: 2.3
> Environment: centos 7, hbase 0.98
> Reporter: hakim
> Priority: Critical
>
> Testing with the following site: https://www.ouedkniss.com, nutch only parse
> links that does contain the base url.
> Tried tika as parser, tried to update db.max.outlinks.per.page to -1, tried
> practically every comments about detecting all the links, doubted urlfilter
> or regex-normalizer so it was disabled but having the same results.
> each time I rebuild nutch and test the parser, it gives the same urls count
> arround 378.
> Can somebody help out to fix this.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
