Hello Dave,

You have both TikaParser and HtmlParser enabled. This probably means you never 
use HtmlParser but always TikaParser. You can instruct Nutch via 
parse-plugins.xml which Parser impl. to choose based on MIME-type. If you 
select HtmlParser for html and xhtml, Nutch should use HtmlParser instead.

Regards,
Markus
 
-----Original message-----
> From:Dave Beckstrom <dbeckst...@collectivefls.com>
> Sent: Wednesday 9th October 2019 22:10
> To: user@nutch.apache.org
> Subject: Nutch excludeNodes Patch
> 
> Hi Everyone!
> 
> 
> We are running Nutch 1.15.
> 
> We are trying to implement the nutch-585-excludeNodes.patch described on:
> https://issues.apache.org/jira/browse/NUTCH-585
> 
> It's acting like it's not running.  We don't get an error when the crawl
> runs, no errors in the hadoop logs, it just doesn't exclude the content
> from the page.
> 
> We installed it in the directory plugins>parse-html
> 
> We added the following to our nutch-site.xml to exclude div id=sidebar
> 
> <property>
>   <name>parser.html.NodesToExclude</name>
>   <value>div;id;sidebar</value>
>   <description>
>   A list of nodes whose content will not be indexed separated by "|".  Use
> this to tell
>   the HTML parser to ignore, for example, site navigation text.
>   Each node has three elements: the first one is the tag name, the second
> one the
>   attribute name, the third one the value of the attribute.
>   Note that nodes with these attributes, and their children, will be
> silently ignored by the parser
>   so verify the indexed content with Luke to confirm results.
>   </description>
> </property>
> 
> Here is our plugin.includes property from nutch-site.xml
> 
>  <property>
>   <name>plugin.includes</name>
> 
> <value>exchange-jexl|protocol-http|urlfilter-regex|parse-(html|tika|metatags)|index-(basic|anchor|metadata)|query-(basic|site|url)|response-(json|xml)|summary-basic|scoring-opic|indexer-solr|urlnormalizer-(pass|regex|basic)</value>
>   <description> plugins
>   </description>
>  </property>
> 
> One question I have is  would having Tika configured in nutch-site.xml like
> the following  cause any problems with the parse-html plugin not running?
> 
> <property>
>   <name>tika.extractor</name>
>   <value>boilerpipe</value>
>   <description>
>   Which text extraction algorithm to use. Valid values are: boilerpipe or
> none.
>   </description>
> </property>
>  <!-- DMB added -->
> <property>
>   <name>tika.extractor.boilerpipe.algorithm</name>
>   <value>ArticleExtractor</value>
>   <description>
>   Which Boilerpipe algorithm to use. Valid values are: DefaultExtractor,
> ArticleExtractor
>   or CanolaExtractor.
>   </description>
> </property>
> 
> We don't have a lot to go on to debug the issue.  The plugin has logic to
> enable logging:
> 
> if (LOG.isTraceEnabled())
> +        LOG.trace("Stripping " + pNode.getNodeName() + "#" +
> idNode.getNodeValue());
> 
> But nothing shows in the log files when we crawl. I
> updated log4j.properties setting these two values to TRACE thinking I had
> to enable trace before the logging would work:
> 
>  log4j.logger.org.apache.nutch.crawl.Crawl=TRACE,cmdstdout
>  log4j.logger.org.apache.nutch.parse.html=TRACE,cmdstdout
> 
> I reran the crawl and no logging occurred and of course the content  we
> didn't want crawled and indexed is still showing up in SOLR.
> 
> I could really use some help and suggestions!
> 
> Thank you!
> 
> Dave Beckstrom
> 
> -- 
> *Fig Leaf Software is now Collective FLS, Inc.*
> *
> *
> *Collective FLS, Inc.* 
> 
> https://www.collectivefls.com/ <https://www.collectivefls.com/> 
> 
> 
> 
> 

Reply via email to