[ https://issues.apache.org/jira/browse/NUTCH-797?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Robert Hohman updated NUTCH-797: -------------------------------- Component/s: parser > parse-tika is not properly constructing URLs when the target begins with a "?" > ------------------------------------------------------------------------------ > > Key: NUTCH-797 > URL: https://issues.apache.org/jira/browse/NUTCH-797 > Project: Nutch > Issue Type: Bug > Components: parser > Affects Versions: 1.1 > Environment: Win 7, Java(TM) SE Runtime Environment (build > 1.6.0_16-b01) > Also repro's on RHEL and java 1.4.2 > Reporter: Robert Hohman > Priority: Minor > Attachments: pureQueryUrl.patch > > > This is my first bug and patch on nutch, so apologies if I have not provided > enough detail. > In crawling the page at > http://careers3.accenture.com/Careers/ASPX/Search.aspx?co=0&sk=0 there are > links in the page that look like this: > <a href="?co=0&sk=0&p=2&pi=1">2</a></td><td><a > href="?co=0&sk=0&p=3&pi=1">3</a> > in org.apache.nutch.parse.tika.DOMContentUtils rev 916362 (trunk), as > getOutlinks looks for links, it comes across this link, and constucts a new > url with a base URL class built from > "http://careers3.accenture.com/Careers/ASPX/Search.aspx?co=0&sk=0", and a > target of "?co=0&sk=0&p=2&pi=1" > The URL class, per RFC 3986 at > http://labs.apache.org/webarch/uri/rfc/rfc3986.html#relative-merge, defines > how to merge these two, and per the RFC, the URL class merges these to: > http://careers3.accenture.com/Careers/ASPX/?co=0&sk=0&p=2&pi=1 > because the RFC explicitly states that the rightmost url segment (the > Search.aspx in this case) should be ripped off before combining. > While this is compliant with the RFC, it means the URLs which are created for > the next round of fetching are incorrect. Modern browsers seem to handle > this case (I checked IE8 and Firefox 3.5), so I'm guessing this is an obscure > exception or handling of what is a poorly formed url on accenture's part. > I have fixed this by modifying DOMContentUtils to look for the case where a ? > begins the target, and then pulling the rightmost component out of the base > and inserting it into the target before the ?, so the target in this example > becomes: > Search.aspx?co=0&sk=0&p=2&pi=1 > The URL class then properly constructs the new url as: > http://careers3.accenture.com/Careers/ASPX/Search.aspx?co=0&sk=0&p=2&pi=1 > If it is agreed that this solution works, I believe the other html parsers in > nutch would need to be modified in a similar way. > Can I get feedback on this proposed solution? Specifically I'm worried about > unforeseen side effects. > Much thanks > Here is the patch info: > Index: > src/plugin/parse-tika/src/java/org/apache/nutch/parse/tika/DOMContentUtils.java > =================================================================== > --- > src/plugin/parse-tika/src/java/org/apache/nutch/parse/tika/DOMContentUtils.java > (revision 916362) > +++ > src/plugin/parse-tika/src/java/org/apache/nutch/parse/tika/DOMContentUtils.java > (working copy) > @@ -299,6 +299,50 @@ > return false; > } > > + private URL fixURL(URL base, String target) throws MalformedURLException > + { > + // handle params that are embedded into the base url - move them to > target > + // so URL class constructs the new url class properly > + if (base.toString().indexOf(';') > 0) > + return fixEmbeddedParams(base, target); > + > + // handle the case that there is a target that is a pure query. > + // Strictly speaking this is a violation of RFC 2396 section 5.2.2 on > how to assemble > + // URLs but I've seen this in numerous places, for example at > + // http://careers3.accenture.com/Careers/ASPX/Search.aspx?co=0&sk=0 > + // It has urls in the page of the form href="?co=0&sk=0&pg=1", and by > default > + // URL constructs the base+target combo as > + // http://careers3.accenture.com/Careers/ASPX/?co=0&sk=0&pg=1, > incorrectly > + // dropping the Search.aspx target > + // > + // Browsers handle these just fine, they must have an exception > similar to this > + if (target.startsWith("?")) > + { > + return fixPureQueryTargets(base, target); > + } > + > + return new URL(base, target); > + } > + > + private URL fixPureQueryTargets(URL base, String target) throws > MalformedURLException > + { > + if (!target.startsWith("?")) > + return new URL(base, target); > + > + String basePath = base.getPath(); > + String baseRightMost=""; > + int baseRightMostIdx = basePath.lastIndexOf("/"); > + if (baseRightMostIdx != -1) > + { > + baseRightMost = basePath.substring(baseRightMostIdx+1); > + } > + > + if (target.startsWith("?")) > + target = baseRightMost+target; > + > + return new URL(base, target); > + } > + > /** > * Handles cases where the url param information is encoded into the base > * url as opposed to the target. > @@ -400,8 +444,7 @@ > if (target != null && !noFollow && !post) > try { > > - URL url = (base.toString().indexOf(';') > 0) ? > - fixEmbeddedParams(base, target) : new URL(base, target); > + URL url = fixURL(base, target); > outlinks.add(new Outlink(url.toString(), > linkText.toString().trim())); > } catch (MalformedURLException e) { -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.