[
https://issues.apache.org/jira/browse/NUTCH-2032?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14576839#comment-14576839
]
Sebastian Nagel commented on NUTCH-2032:
----------------------------------------
Hi [~betolink], from the point of "intrusiveness" or encapsulation I agree - it
would be better to have all necessary functionality bundled inside a single
plugin. But in doubt performance is more important, esp. at scale. With regard
to indexing raw content these points may impact performance:
* increasing the size of segments by adding raw content twice (in average by
50% or more)
* reading unnecessarily the raw content from segments if raw content isn't
indexed (most users only want the readable text to be indexed)
The last patch for NUTCH-1785 does not impact performance. The solution via a
plugin index-rawcontent does.
Btw., in 2.x plugins can request via the FieldPluggable interface which fields
are read from the web table. That's nice, but in 1.x we don't have such a
functionality.
> Plugin to index the raw content of a readable document.
> --------------------------------------------------------
>
> Key: NUTCH-2032
> URL: https://issues.apache.org/jira/browse/NUTCH-2032
> Project: Nutch
> Issue Type: New Feature
> Components: indexer, parser
> Affects Versions: 1.10
> Reporter: Luis Lopez
> Labels: content, index, index-rawcontent, parser, raw
> Fix For: 1.11
>
>
> This is related to https://issues.apache.org/jira/browse/NUTCH-1785 and
> https://issues.apache.org/jira/browse/NUTCH-1458
> We created a couple plugins to index the raw content of readable documents.
> If we include these plugins in the plugin chain we'll index the raw content
> of a readable document, i.e. XML, HTML, CSV, TXT etc. The index-rawcontent
> plugin is not designed to index binary files, however having the full content
> of an HTML/XML or a CSV document is really critical for some of us.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
