[
https://issues.apache.org/jira/browse/NUTCH-2598?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16920373#comment-16920373
]
Hudson commented on NUTCH-2598:
-------------------------------
SUCCESS: Integrated in Jenkins build Nutch-trunk #3640 (See
[https://builds.apache.org/job/Nutch-trunk/3640/])
NUTCH-2598 URLNormalizerChecker fails on invalid URLs in input - output
(snagel:
[https://github.com/apache/nutch/commit/36c2ce6925a7fc4e4c3f5b6052ee96d491d2e5eb])
* (edit) src/java/org/apache/nutch/net/URLNormalizerChecker.java
> URLNormalizerChecker fails on invalid URLs in input
> ---------------------------------------------------
>
> Key: NUTCH-2598
> URL: https://issues.apache.org/jira/browse/NUTCH-2598
> Project: Nutch
> Issue Type: Bug
> Affects Versions: 1.14
> Reporter: Sebastian Nagel
> Assignee: Sebastian Nagel
> Priority: Minor
> Fix For: 1.16
>
>
> I use the URLNormalizerChecker (urlnormalizer-regex and urlnormalizer-basic)
> to normalize URLs before further processing them. If one of the used
> normalizers throws a MalformedURLException when the
> URLNormalizer.normalize(...) method is called, this isn't caught and causes
> the checker to exit:
> {noformat}
> Exception in thread "main" java.net.MalformedURLException: For input string:
> "???120810002"
> at java.net.URL.<init>(URL.java:627)
> at java.net.URL.<init>(URL.java:490)
> at java.net.URL.<init>(URL.java:439)
> at
> org.apache.nutch.net.urlnormalizer.basic.BasicURLNormalizer.normalize(BasicURLNormalizer.java:100)
> at
> org.apache.nutch.net.URLNormalizers.normalize(URLNormalizers.java:319)
> at
> org.apache.nutch.net.URLNormalizerChecker.process(URLNormalizerChecker.java:75)
> at
> org.apache.nutch.util.AbstractChecker.processStdin(AbstractChecker.java:97)
> at org.apache.nutch.util.AbstractChecker.run(AbstractChecker.java:77)
> at
> org.apache.nutch.net.URLNormalizerChecker.run(URLNormalizerChecker.java:71)
> at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70)
> at
> org.apache.nutch.net.URLNormalizerChecker.main(URLNormalizerChecker.java:80)
> Caused by: java.lang.NumberFormatException: For input string: "???120810002"
> at
> java.lang.NumberFormatException.forInputString(NumberFormatException.java:65)
> at java.lang.Integer.parseInt(Integer.java:580)
> at java.lang.Integer.parseInt(Integer.java:615)
> at java.net.URLStreamHandler.parseURL(URLStreamHandler.java:222)
> at java.net.URL.<init>(URL.java:622)
> ... 10 more
> {noformat}
> The URLNormalizer interface declares the MalformedURLException, it should be
> caught in the normalizer checker:
> - log the error
> - return/output empty string
--
This message was sent by Atlassian Jira
(v8.3.2#803003)
