[
https://issues.apache.org/jira/browse/NUTCH-2435?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16185755#comment-16185755
]
ASF GitHub Bot commented on NUTCH-2435:
---------------------------------------
maborec commented on a change in pull request #225: NUTCH-2435 - New parameter
"parser.store.text"
URL: https://github.com/apache/nutch/pull/225#discussion_r141856440
##########
File path: src/java/org/apache/nutch/parse/ParseOutputFormat.java
##########
@@ -128,13 +131,18 @@ public void checkOutputSpecs(FileSystem fs, JobConf job)
throws IOException {
.split(" *, *");
// textOut Options
- Option tKeyClassOpt = (Option) MapFile.Writer.keyClass(Text.class);
- org.apache.hadoop.io.SequenceFile.Writer.Option tValClassOpt =
SequenceFile.Writer.valueClass(ParseText.class);
- org.apache.hadoop.io.SequenceFile.Writer.Option tProgressOpt =
SequenceFile.Writer.progressable(progress);
- org.apache.hadoop.io.SequenceFile.Writer.Option tCompOpt =
SequenceFile.Writer.compression(CompressionType.RECORD);
+ final MapFile.Writer textOut;
+ if (storeText) {
+ Option tKeyClassOpt = (Option) MapFile.Writer.keyClass(Text.class);
+ org.apache.hadoop.io.SequenceFile.Writer.Option tValClassOpt =
SequenceFile.Writer.valueClass(ParseText.class);
+ org.apache.hadoop.io.SequenceFile.Writer.Option tProgressOpt =
SequenceFile.Writer.progressable(progress);
+ org.apache.hadoop.io.SequenceFile.Writer.Option tCompOpt =
SequenceFile.Writer.compression(CompressionType.RECORD);
- final MapFile.Writer textOut = new MapFile.Writer(job, text,
+ textOut = new MapFile.Writer(job, text,
Review comment:
Format applied
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
[email protected]
> New configuration allowing to choose whether to store 'parse_text' directory
> or not.
> ------------------------------------------------------------------------------------
>
> Key: NUTCH-2435
> URL: https://issues.apache.org/jira/browse/NUTCH-2435
> Project: Nutch
> Issue Type: New Feature
> Components: parser
> Affects Versions: 1.13
> Environment: Apach Nutch 1.13
> Reporter: Marcos Bori
>
> Whenever a page is parsed, one of the outputs is the directory 'parse_text'.
> It is intended to be used at the indexing phase so the page can be searched
> from a search engine such as Solr.
> In my special crawling case, I don't need to index the page contents.
> Therefore, creating and filing the 'parse_text' is not required for me. To
> optimize performance, I don't want the crawler to store this information to
> the filesystem.
> I propose a new parameter "parser.store.text" allowing to choose whether to
> store 'parse_text' directory or not. Its default value, of course, is "true".
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
