You are sure the patch works? You get different text output with
tika.use_boilerpipe enabled and disabled?
-----Original message-----
> From:Joe Zhang <[email protected]>
> Sent: Wednesday 7th August 2013 20:12
> To: user <[email protected]>
> Subject: Boilerplate removal
>
> I'm having the following in my nutchsite.xml. Yet the boilerplate removal
> isn't quite successful. A lot of webpages (from reputable sources such as
> reuters.com) come with sidepanes and other junks that were not removed. Any
> suggestions from the experts?
>
> <name>plugin.includes</name>
>
> <value>protocol-http|urlfilter-regex|parse-(html|tika|metatags)|index-(basic|anchor|metadata|more)|scoring-opic|urlnormalizer-(pass|regex|basic)</value>
> <description>Regular expression naming plugin directory names to
> include. Any plugin not matching this expression is excluded.
> In any case you need at least include the nutch-extensionpoints plugin. By
> default Nutch includes crawling just HTML and plain text via HTTP,
> and basic indexing and search plugins. In order to use HTTPS please enable
> protocol-httpclient, but be aware of possible intermittent problems with
> the
> underlying commons-httpclient library.
> </description>
> </property>
> <!-- tika properties to use BoilerPipe, according to Marcus Jelsma -->
> <property>
> <name>tika.use_boilerpipe</name>
> <value>true</value>
> </property>
> <property>
> <name>tika.boilerpipe.extractor</name>
> <value>ArticleExtractor</value>
> </property>
>
