Shaya, You might want to look at the content's of bin/nutch.
When I want to to test the parser, I call ./bin/nutch org.apache.nutch.parse.ParserChecker http://domain.com/document.html. One point to note is that bin/nutch load the configuration before call the classes. you would need the configuration set properly for most of the command to work. Hope it helps, Ye On Mon, Aug 27, 2012 at 12:23 AM, Markus Jelsma <[email protected]>wrote: > See: https://issues.apache.org/jira/browse/NUTCH-961 > > > > -----Original message----- > > From:Shaya Potter <[email protected]> > > Sent: Sun 26-Aug-2012 17:59 > > To: [email protected] > > Subject: Re: running main() in plugins? > > > > It could be the "magic" (i.e. analysis) that Nutch is doing in the > > background gets rid of most of the cruft, I'm just playing around on my > > own trying to see how I can get the best text to analyze, and in many > > cases, there's a lot of cruft and I was wondering if Nutch did anything > > to remove said cruft (headers, footers, sidebars....) > > > > what I'm doing now for my experiments is relatively heavyweight, but > > I'm, applying the readability algorithm to web pages before I index them > > into my a lucene database. probably not the best idea for nutch though. > > > > With that said, if Nutch is doing more processing than a jsoup > > Document.text() operation, the question is why? (some might be obvious, > > metadata, getting outbound links) > > > > On 08/26/2012 08:55 AM, Lewis John Mcgibbney wrote: > > > Hi Shaya, > > > > > > Can you elaborate? The plugin has been around for a good while. If you > > > have suggestions to improve they are very welcome. > > > > > > Thanks > > > > > > On Sun, Aug 26, 2012 at 1:41 PM, Shaya Potter <[email protected]> > wrote: > > >> ok, so it seems that Nutch isn't doing much different (at least from a > > >> smattering of tests I've done) than Jsoup's Document.text() ability > (from > > >> what I can tell at least, perhaps only some issues with spacing > between > > >> elements). > > >> > > >> On 08/26/2012 06:28 AM, Lewis John Mcgibbney wrote: > > >>> > > >>> You can easily run any plugin from the terminal using > > >>> > > >>> ./bin/nutch plugin > > >>> > > >>> in the case of the HtmlParser main() method you would want to do > > >>> > > >>> ./bin/nutch plugin parse-html org.apache.nutch.parse.html.HtmlParser > > >>> $pathToLocalFile > > >>> > > >>> You have actually identified an improvement which we could do with > > >>> having in the main() method for this class e.g. > > >>> > > >>> 1) When the arguments are not correctly specified it should print a > > >>> usage message to std out explaining the correct plugin usage as with > > >>> more or less every other plugin. Currently we just get a nasty stack > > >>> like the following > > >>> > > >>> Exception in thread "main" > java.lang.reflect.InvocationTargetException > > >>> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native > Method) > > >>> at > > >>> > sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39) > > >>> at > > >>> > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) > > >>> at java.lang.reflect.Method.invoke(Method.java:597) > > >>> at > > >>> > org.apache.nutch.plugin.PluginRepository.main(PluginRepository.java:421) > > >>> Caused by: java.io.FileNotFoundException: > > >>> http:/www.trancearoundtheworld.com (No such file or directory) > > >>> at java.io.FileInputStream.open(Native Method) > > >>> at java.io.FileInputStream.<init>(FileInputStream.java:120) > > >>> at > > >>> org.apache.nutch.parse.html.HtmlParser.main(HtmlParser.java:274) > > >>> ... 5 more > > >>> > > >>> 2) The plugin main method only enables you to parse local files an > > >>> improvement would be to add functionality similar to the > parserchecker > > >>> as highlighted by Sourajit > > >>> > > >>> If you wish to add these functions then please open a Jira issue, the > > >>> contribution would be great. > > >>> > > >>> Thanks > > >>> > > >>> Lewis > > >>> > > >>> On Sun, Aug 26, 2012 at 4:18 AM, Shaya Potter <[email protected]> > wrote: > > >>>> > > >>>> I'm trying to run the main function in HtmlParser (just to see test > how > > >>>> Nutch's parser works compared to others) and I can't see to figure > out > > >>>> how > > >>>> to get it to run. > > >>>> > > >>>> > > >>>> > http://svn.apache.org/viewvc/nutch/branches/branch-1.5.1/src/plugin/parse-html/src/java/org/apache/nutch/parse/html/HtmlParser.java?revision=1356339&view=markup > > >>>> > > >>>> when I run it naively, I get an error > > >>>> > > >>>> Exception in thread "main" java.lang.RuntimeException: > > >>>> org.apache.nutch.parse.HtmlParseFilter not found. > > >>>> at > > >>>> > org.apache.nutch.parse.HtmlParseFilters.<init>(HtmlParseFilters.java:55) > > >>>> > > >>>> in looking at HtmlParseFilters, I see that it throws the runtime > > >>>> exception > > >>>> if it can't find any HtmlParseFilter classes, however, I can't seem > to > > >>>> figure out how to make it able to find them (I see the jar's in the > > >>>> plugins > > >>>> dir, but do they have to be registered? could the main() in > HtmlParser > > >>>> ever > > >>>> work as is? > > >>>> > > >>>> any pointers would be appreciated. > > >>>> > > >>>> thanks. > > >>>> > > >>> > > >>> > > >>> > > >> > > > > > > > > > > > >
