[
https://issues.apache.org/jira/browse/NUTCH-585?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
N. Hira updated NUTCH-585:
--------------------------
Attachment: nutch-585-jostens-excludeDIVs.patch
We use Solr/Nutch on our corporate web site and are very happy with the
results. Thank you. We have struggled with something similar to NUTCH-585 for
a few months now.
Although it is different from the original intent, here's a quick/short patch
that might help get this feature going again.
h4.Intended use:
- Let's assume you're crawling a set of internal web sites and would like to
exclude certain HTML fragments (from indexing) like the navigation and other
common content.
- If these fragments are contained in DIVs with IDs like "menuNav",
"footerNav", etc., then you can now add a new property to nutch-site.xml to
exclude these DIVs.
- If you don't set this property, the normal behavior remains (backward
compatible)
{code:xml}
<property>
<name>parser.html.divIDsToExclude</name
<value>account_menu_container,footer_menu_container,legal,main_menu_container</value>
<description>
A comma-delimited list of DIV IDs whose content will not be indexed. Use
this to tell
the HTML parser to ignore, for example, site navigation text.
Note that DIVs with these IDs, and their children, will be silently ignored
by the parser
so verify the indexed content with Luke to confirm results.
</description>
</property>
{code}
h4.Inclusion/growth:
- This code was written against nutch 1.2 and is backward compatible in that
the new behavior is only present if configured.
- In future, it might be good to have different "strategy patterns" for how
exclusions are determined; some might need algorithmic detection (whole web
crawls), others might prefer jquery-selectors for HTML fragments, etc.
Best regards,
-h
Hira, N.R. (Jostens, Inc.)
> [PARSE-HTML plugin] Block certain parts of HTML code from being indexed
> -----------------------------------------------------------------------
>
> Key: NUTCH-585
> URL: https://issues.apache.org/jira/browse/NUTCH-585
> Project: Nutch
> Issue Type: Improvement
> Affects Versions: 0.9.0
> Environment: All operating systems
> Reporter: Andrea Spinelli
> Priority: Minor
> Attachments: nutch-585-jostens-excludeDIVs.patch
>
>
> We are using nutch to index our own web sites; we would like not to index
> certain parts of our pages, because we know they are not relevant (for
> instance, there are several links to change the background color) and
> generate spurious matches.
> We have modified the plugin so that it ignores HTML code between certain HTML
> comments, like
> <!-- START-IGNORE -->
> ... ignored part ...
> <!-- STOP-IGNORE -->
> We feel this might be useful to someone else, maybe factorizing the comment
> strings as constants in the configuration files (say parser.html.ignore.start
> and parser.html.ignore.stop in nutch-site.xml).
> We are almost ready to contribute our code snippet. Looking forward for any
> expression of interest - or for an explanation why waht we are doing is
> plain wrong!
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
