didn't know about boilerplate before, coded up a quick test (think this
is correct, but please point out errors if I'm doing something boneheaded)
static public JResult tika_extract(String htmltext) {
InputStream input = new ByteArrayInputStream(htmltext.getBytes());
BoilerpipeContentHandler handler = new
BoilerpipeContentHandler(new BodyContentHandler());
//BodyContentHandler handler = new BodyContentHandler();
Metadata metadata = new Metadata();
try {
new HtmlParser().parse(input, handler, metadata, new
ParseContext());
} catch (IOException e) {
// TODO Auto-generated catch block
e.printStackTrace();
} catch (SAXException e) {
// TODO Auto-generated catch block
e.printStackTrace();
} catch (TikaException e) {
// TODO Auto-generated catch block
e.printStackTrace();
}
return handler.toTextDocument().getContent();
}
and it seems to give reasonable results (sometimes a little better than
readability, sometimes a little worse), but the major problem it throws
stack exceptions enough in cyberneko enough (i.e. for many, if not all,
nytimes pages I pointed it at).
On 08/26/2012 12:23 PM, Markus Jelsma wrote:
See: https://issues.apache.org/jira/browse/NUTCH-961
-----Original message-----
From:Shaya Potter <[email protected]>
Sent: Sun 26-Aug-2012 17:59
To: [email protected]
Subject: Re: running main() in plugins?
It could be the "magic" (i.e. analysis) that Nutch is doing in the
background gets rid of most of the cruft, I'm just playing around on my
own trying to see how I can get the best text to analyze, and in many
cases, there's a lot of cruft and I was wondering if Nutch did anything
to remove said cruft (headers, footers, sidebars....)
what I'm doing now for my experiments is relatively heavyweight, but
I'm, applying the readability algorithm to web pages before I index them
into my a lucene database. probably not the best idea for nutch though.
With that said, if Nutch is doing more processing than a jsoup
Document.text() operation, the question is why? (some might be obvious,
metadata, getting outbound links)
On 08/26/2012 08:55 AM, Lewis John Mcgibbney wrote:
Hi Shaya,
Can you elaborate? The plugin has been around for a good while. If you
have suggestions to improve they are very welcome.
Thanks
On Sun, Aug 26, 2012 at 1:41 PM, Shaya Potter <[email protected]> wrote:
ok, so it seems that Nutch isn't doing much different (at least from a
smattering of tests I've done) than Jsoup's Document.text() ability (from
what I can tell at least, perhaps only some issues with spacing between
elements).
On 08/26/2012 06:28 AM, Lewis John Mcgibbney wrote:
You can easily run any plugin from the terminal using
./bin/nutch plugin
in the case of the HtmlParser main() method you would want to do
./bin/nutch plugin parse-html org.apache.nutch.parse.html.HtmlParser
$pathToLocalFile
You have actually identified an improvement which we could do with
having in the main() method for this class e.g.
1) When the arguments are not correctly specified it should print a
usage message to std out explaining the correct plugin usage as with
more or less every other plugin. Currently we just get a nasty stack
like the following
Exception in thread "main" java.lang.reflect.InvocationTargetException
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at
org.apache.nutch.plugin.PluginRepository.main(PluginRepository.java:421)
Caused by: java.io.FileNotFoundException:
http:/www.trancearoundtheworld.com (No such file or directory)
at java.io.FileInputStream.open(Native Method)
at java.io.FileInputStream.<init>(FileInputStream.java:120)
at
org.apache.nutch.parse.html.HtmlParser.main(HtmlParser.java:274)
... 5 more
2) The plugin main method only enables you to parse local files an
improvement would be to add functionality similar to the parserchecker
as highlighted by Sourajit
If you wish to add these functions then please open a Jira issue, the
contribution would be great.
Thanks
Lewis
On Sun, Aug 26, 2012 at 4:18 AM, Shaya Potter <[email protected]> wrote:
I'm trying to run the main function in HtmlParser (just to see test how
Nutch's parser works compared to others) and I can't see to figure out
how
to get it to run.
http://svn.apache.org/viewvc/nutch/branches/branch-1.5.1/src/plugin/parse-html/src/java/org/apache/nutch/parse/html/HtmlParser.java?revision=1356339&view=markup
when I run it naively, I get an error
Exception in thread "main" java.lang.RuntimeException:
org.apache.nutch.parse.HtmlParseFilter not found.
at
org.apache.nutch.parse.HtmlParseFilters.<init>(HtmlParseFilters.java:55)
in looking at HtmlParseFilters, I see that it throws the runtime
exception
if it can't find any HtmlParseFilter classes, however, I can't seem to
figure out how to make it able to find them (I see the jar's in the
plugins
dir, but do they have to be registered? could the main() in HtmlParser
ever
work as is?
any pointers would be appreciated.
thanks.
