bug in DomContentUtils.shouldThrowAwayLink?
-------------------------------------------
Key: NUTCH-690
URL: https://issues.apache.org/jira/browse/NUTCH-690
Project: Nutch
Issue Type: Bug
Affects Versions: 0.9.0
Reporter: Peter Sparks
Priority: Minor
I found a potential bug in DomContentUtils.shouldThrowAwayLink. It returns true
for the 5 links at the top of the home page for www.aksteel.com. Here are the
links in the source:
<a id="Search" href="/search/default.aspx"></a>
<a id="Investor" style="height: 15px;"
href="/investor_information/"></a>
<a id="Markets" href="/markets_products/"></a>
<a id="Production" href="/production_facilities/"></a>
<a id="News" href="/news/"></a>
Perhaps I am just ignorant of what this function is supposed to do but
returning true for these 5 links on that site make that site impossible to
crawl.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
