ok, so it seems that Nutch isn't doing much different (at least from a
smattering of tests I've done) than Jsoup's Document.text() ability
(from what I can tell at least, perhaps only some issues with spacing
between elements).
On 08/26/2012 06:28 AM, Lewis John Mcgibbney wrote:
You can easily run any plugin from the terminal using
./bin/nutch plugin
in the case of the HtmlParser main() method you would want to do
./bin/nutch plugin parse-html org.apache.nutch.parse.html.HtmlParser
$pathToLocalFile
You have actually identified an improvement which we could do with
having in the main() method for this class e.g.
1) When the arguments are not correctly specified it should print a
usage message to std out explaining the correct plugin usage as with
more or less every other plugin. Currently we just get a nasty stack
like the following
Exception in thread "main" java.lang.reflect.InvocationTargetException
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at
org.apache.nutch.plugin.PluginRepository.main(PluginRepository.java:421)
Caused by: java.io.FileNotFoundException:
http:/www.trancearoundtheworld.com (No such file or directory)
at java.io.FileInputStream.open(Native Method)
at java.io.FileInputStream.<init>(FileInputStream.java:120)
at org.apache.nutch.parse.html.HtmlParser.main(HtmlParser.java:274)
... 5 more
2) The plugin main method only enables you to parse local files an
improvement would be to add functionality similar to the parserchecker
as highlighted by Sourajit
If you wish to add these functions then please open a Jira issue, the
contribution would be great.
Thanks
Lewis
On Sun, Aug 26, 2012 at 4:18 AM, Shaya Potter <[email protected]> wrote:
I'm trying to run the main function in HtmlParser (just to see test how
Nutch's parser works compared to others) and I can't see to figure out how
to get it to run.
http://svn.apache.org/viewvc/nutch/branches/branch-1.5.1/src/plugin/parse-html/src/java/org/apache/nutch/parse/html/HtmlParser.java?revision=1356339&view=markup
when I run it naively, I get an error
Exception in thread "main" java.lang.RuntimeException:
org.apache.nutch.parse.HtmlParseFilter not found.
at
org.apache.nutch.parse.HtmlParseFilters.<init>(HtmlParseFilters.java:55)
in looking at HtmlParseFilters, I see that it throws the runtime exception
if it can't find any HtmlParseFilter classes, however, I can't seem to
figure out how to make it able to find them (I see the jar's in the plugins
dir, but do they have to be registered? could the main() in HtmlParser ever
work as is?
any pointers would be appreciated.
thanks.
