[
https://issues.apache.org/jira/browse/NUTCH-585?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13486041#comment-13486041
]
Roberto Gardenier commented on NUTCH-585:
-----------------------------------------
I have compiled nutch 1.5.1 with the provided plugin and used the configuration
as described above. This all without success.
Could anyone assist me on troubleshooting ?
Nutch crawls and SOLR indexes with success but the content field still includes
content of which are supposed to be blacklisted.
Steps:
1. Patched Nutch 1.5.1. with above blacklist_whitelist_plugin.patch
2. Enabled the plugin in nutch-default.xml plugin.includes:
index-blacklist-whitelist.
3. Added the new field strippedContent to schema.xml (both nutch and solr) <!--
fields for the blacklist/whitelist plugin --> <field name="strippedContent"
type="text" stored="true" indexed="true"/>.
4. Configured parser.html.blacklist to blacklist "div.kruimelspoor" in
nutch-default.xml.
I pointed nutch at my site and fired it. I dont get warnings/errors or any kind
of showstoppers, the crawling goes well and the index is filled. But still with
everything inside div.kruimelspoor.
> [PARSE-HTML plugin] Block certain parts of HTML code from being indexed
> -----------------------------------------------------------------------
>
> Key: NUTCH-585
> URL: https://issues.apache.org/jira/browse/NUTCH-585
> Project: Nutch
> Issue Type: Improvement
> Affects Versions: 0.9.0
> Environment: All operating systems
> Reporter: Andrea Spinelli
> Assignee: Markus Jelsma
> Priority: Minor
> Fix For: 1.6
>
> Attachments: blacklist_whitelist_plugin.patch,
> nutch-585-excludeNodes.patch, nutch-585-jostens-excludeDIVs.patch
>
>
> We are using nutch to index our own web sites; we would like not to index
> certain parts of our pages, because we know they are not relevant (for
> instance, there are several links to change the background color) and
> generate spurious matches.
> We have modified the plugin so that it ignores HTML code between certain HTML
> comments, like
> <!-- START-IGNORE -->
> ... ignored part ...
> <!-- STOP-IGNORE -->
> We feel this might be useful to someone else, maybe factorizing the comment
> strings as constants in the configuration files (say parser.html.ignore.start
> and parser.html.ignore.stop in nutch-site.xml).
> We are almost ready to contribute our code snippet. Looking forward for any
> expression of interest - or for an explanation why waht we are doing is
> plain wrong!
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
