[
https://issues.apache.org/jira/browse/NUTCH-984?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Markus Jelsma updated NUTCH-984:
--------------------------------
Description:
For some reason using parse-tika a crawl just wouldn't dive into some website
news archive. The paging through the news archive is done with simple anchors:
<div class="page active">1</div> <a href="/nieuws/overzicht/1/"><div
class="page">2</div> </a> <a href="/nieuws/overzicht/2/"><div
class="page">3</div> </a>
I added some logging to DOMContentUtils:
2011-04-18 18:26:09,788 INFO tika.DOMContentUtils - Throw away link:
http://www.site.nl/nieuws/overzicht/1/
2011-04-18 18:26:09,788 INFO tika.DOMContentUtils - Throw away link:
http://www.site.nl/nieuws/overzicht/2/
2011-04-18 18:26:09,788 INFO tika.DOMContentUtils - Throw away link:
http://www.site.nl/nieuws/overzicht/3/
...
Now, this is rather funky. The code for private boolean
shouldThrowAwayLink(Node node, NodeList children, int childLen, LinkParams
params) is the same for parse-html and parse-tika. I also tested the two
parsers between versions 1.2 and 1.3 for the following URL.
http://news.bbc.co.uk/2/hi/europe/country_profiles/1154019.stm
1.2 - parse-tika: 196
1.2 - parse-html: 296
1.3 - parse-tika: 279
1.3 - parse-html: 296
Something clearly improved in 1.3 but not generating the remaining URL's are a
blocker for parse-tika in my case. Relevant configurations are the same
parser.html.outlinks.ignore_tags is not being used. Testing has been done with
ParserChecker only.
was:
For some reason using parse-tika a crawl just wouldn't dive into some website
news archive. The paging through the news archive is done with simple anchors:
<div class="page active">1</div> <a href="/nieuws/overzicht/1/"><div
class="page">2</div> </a> <a href="/nieuws/overzicht/2/"><div
class="page">3</div> </a>
I added some logging to DOMContentUtils:
2011-04-18 18:26:09,788 INFO tika.DOMContentUtils - Throw away link:
http://www.site.nl/nieuws/overzicht/1/
2011-04-18 18:26:09,788 INFO tika.DOMContentUtils - Throw away link:
http://www.site.nl/nieuws/overzicht/2/
2011-04-18 18:26:09,788 INFO tika.DOMContentUtils - Throw away link:
http://www.site.nl/nieuws/overzicht/3/
...
Now, this is rather funky. The code for private boolean
shouldThrowAwayLink(Node node, NodeList children, int childLen, LinkParams
params) is the same for parse-html and parse-tika. I also tested the two
parsers between versions 1.2 and 1.3 for the following URL.
http://news.bbc.co.uk/2/hi/europe/country_profiles/1154019.stm
1.2 - parse-tika: 196
1.2 - parse-html: 296
1.3 - parse-tika: 279
1.3 - parse-html: 296
Something clearly improved in 1.3 but not generating the remaining URL's are a
blocker for parse-tika in my case.
> Parse-tika throws some URL's away
> ---------------------------------
>
> Key: NUTCH-984
> URL: https://issues.apache.org/jira/browse/NUTCH-984
> Project: Nutch
> Issue Type: Bug
> Components: parser
> Affects Versions: 1.3, 2.0
> Reporter: Markus Jelsma
> Priority: Critical
> Fix For: 1.3, 2.0
>
>
> For some reason using parse-tika a crawl just wouldn't dive into some website
> news archive. The paging through the news archive is done with simple anchors:
> <div class="page active">1</div> <a href="/nieuws/overzicht/1/"><div
> class="page">2</div> </a> <a href="/nieuws/overzicht/2/"><div
> class="page">3</div> </a>
> I added some logging to DOMContentUtils:
> 2011-04-18 18:26:09,788 INFO tika.DOMContentUtils - Throw away link:
> http://www.site.nl/nieuws/overzicht/1/
> 2011-04-18 18:26:09,788 INFO tika.DOMContentUtils - Throw away link:
> http://www.site.nl/nieuws/overzicht/2/
> 2011-04-18 18:26:09,788 INFO tika.DOMContentUtils - Throw away link:
> http://www.site.nl/nieuws/overzicht/3/
> ...
> Now, this is rather funky. The code for private boolean
> shouldThrowAwayLink(Node node, NodeList children, int childLen, LinkParams
> params) is the same for parse-html and parse-tika. I also tested the two
> parsers between versions 1.2 and 1.3 for the following URL.
> http://news.bbc.co.uk/2/hi/europe/country_profiles/1154019.stm
> 1.2 - parse-tika: 196
> 1.2 - parse-html: 296
> 1.3 - parse-tika: 279
> 1.3 - parse-html: 296
> Something clearly improved in 1.3 but not generating the remaining URL's are
> a blocker for parse-tika in my case. Relevant configurations are the same
> parser.html.outlinks.ignore_tags is not being used. Testing has been done
> with ParserChecker only.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
