[jira] [Commented] (NUTCH-2435) New configuration allowing to choose whether to store 'parse_text' directory or not.
[ https://issues.apache.org/jira/browse/NUTCH-2435?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16211768#comment-16211768 ] ASF GitHub Bot commented on NUTCH-2435: --- sebastian-nagel commented on issue #225: NUTCH-2435 - New parameter "parser.store.text" URL: https://github.com/apache/nutch/pull/225#issuecomment-338043379 Thanks, @maborec! Everything ok, just bussy the last time. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org > New configuration allowing to choose whether to store 'parse_text' directory > or not. > > > Key: NUTCH-2435 > URL: https://issues.apache.org/jira/browse/NUTCH-2435 > Project: Nutch > Issue Type: New Feature > Components: parser >Affects Versions: 1.13 > Environment: Apach Nutch 1.13 >Reporter: Marcos Bori > > Whenever a page is parsed, one of the outputs is the directory 'parse_text'. > It is intended to be used at the indexing phase so the page can be searched > from a search engine such as Solr. > In my special crawling case, I don't need to index the page contents. > Therefore, creating and filing the 'parse_text' is not required for me. To > optimize performance, I don't want the crawler to store this information to > the filesystem. > I propose a new parameter "parser.store.text" allowing to choose whether to > store 'parse_text' directory or not. Its default value, of course, is "true". -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (NUTCH-2435) New configuration allowing to choose whether to store 'parse_text' directory or not.
[ https://issues.apache.org/jira/browse/NUTCH-2435?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16211766#comment-16211766 ] ASF GitHub Bot commented on NUTCH-2435: --- sebastian-nagel closed pull request #225: NUTCH-2435 - New parameter "parser.store.text" URL: https://github.com/apache/nutch/pull/225 This is a PR merged from a forked repository. As GitHub hides the original diff on merge, it is displayed below for the sake of provenance: As this is a foreign pull request (from a fork), the diff is supplied below (as it won't show otherwise due to GitHub magic): diff --git a/conf/nutch-default.xml b/conf/nutch-default.xml index c406907c5..587386140 100644 --- a/conf/nutch-default.xml +++ b/conf/nutch-default.xml @@ -1372,6 +1372,13 @@ CAUTION: Set the parser.timeout to -1 or a bigger value than 30, when using this + + parser.store.text + true + If true (default value), parser will store parse text (parse_text directory within the segment). + + + > Key: NUTCH-2435 > URL: https://issues.apache.org/jira/browse/NUTCH-2435 > Project: Nutch > Issue Type: New Feature > Components: parser >Affects Versions: 1.13 > Environment: Apach Nutch 1.13 >Reporter: Marcos Bori > > Whenever a page is parsed, one of the outputs is the directory 'parse_text'. > It is intended to be used at the indexing phase so the page can be searched > from a search engine such as Solr. > In my special crawling case, I don't need to index the page contents. > Therefore, creating and filing the 'parse_text' is not required for me. To > optimize performance, I don't want the crawler to store this information to > the filesystem. > I propose a new parameter "parser.store.text" allowing to choose whether to > store 'parse_text' directory or not. Its default value, of course, is "true". -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (NUTCH-2435) New configuration allowing to choose whether to store 'parse_text' directory or not.
[ https://issues.apache.org/jira/browse/NUTCH-2435?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16192600#comment-16192600 ] ASF GitHub Bot commented on NUTCH-2435: --- maborec commented on issue #225: NUTCH-2435 - New parameter "parser.store.text" URL: https://github.com/apache/nutch/pull/225#issuecomment-334393715 Hi @sebastian-nagel, I can see you approved the changes some days ago. Do you plan to merge them into the repository? Is there anything missing from my side to proceed? Thanks! M This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org > New configuration allowing to choose whether to store 'parse_text' directory > or not. > > > Key: NUTCH-2435 > URL: https://issues.apache.org/jira/browse/NUTCH-2435 > Project: Nutch > Issue Type: New Feature > Components: parser >Affects Versions: 1.13 > Environment: Apach Nutch 1.13 >Reporter: Marcos Bori > > Whenever a page is parsed, one of the outputs is the directory 'parse_text'. > It is intended to be used at the indexing phase so the page can be searched > from a search engine such as Solr. > In my special crawling case, I don't need to index the page contents. > Therefore, creating and filing the 'parse_text' is not required for me. To > optimize performance, I don't want the crawler to store this information to > the filesystem. > I propose a new parameter "parser.store.text" allowing to choose whether to > store 'parse_text' directory or not. Its default value, of course, is "true". -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (NUTCH-2435) New configuration allowing to choose whether to store 'parse_text' directory or not.
[ https://issues.apache.org/jira/browse/NUTCH-2435?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16185800#comment-16185800 ] ASF GitHub Bot commented on NUTCH-2435: --- sebastian-nagel commented on issue #225: NUTCH-2435 - New parameter "parser.store.text" URL: https://github.com/apache/nutch/pull/225#issuecomment-333122656 +1 This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org > New configuration allowing to choose whether to store 'parse_text' directory > or not. > > > Key: NUTCH-2435 > URL: https://issues.apache.org/jira/browse/NUTCH-2435 > Project: Nutch > Issue Type: New Feature > Components: parser >Affects Versions: 1.13 > Environment: Apach Nutch 1.13 >Reporter: Marcos Bori > > Whenever a page is parsed, one of the outputs is the directory 'parse_text'. > It is intended to be used at the indexing phase so the page can be searched > from a search engine such as Solr. > In my special crawling case, I don't need to index the page contents. > Therefore, creating and filing the 'parse_text' is not required for me. To > optimize performance, I don't want the crawler to store this information to > the filesystem. > I propose a new parameter "parser.store.text" allowing to choose whether to > store 'parse_text' directory or not. Its default value, of course, is "true". -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (NUTCH-2435) New configuration allowing to choose whether to store 'parse_text' directory or not.
[
https://issues.apache.org/jira/browse/NUTCH-2435?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16185755#comment-16185755
]
ASF GitHub Bot commented on NUTCH-2435:
---
maborec commented on a change in pull request #225: NUTCH-2435 - New parameter
"parser.store.text"
URL: https://github.com/apache/nutch/pull/225#discussion_r141856440
##
File path: src/java/org/apache/nutch/parse/ParseOutputFormat.java
##
@@ -128,13 +131,18 @@ public void checkOutputSpecs(FileSystem fs, JobConf job)
throws IOException {
.split(" *, *");
// textOut Options
-Option tKeyClassOpt = (Option) MapFile.Writer.keyClass(Text.class);
-org.apache.hadoop.io.SequenceFile.Writer.Option tValClassOpt =
SequenceFile.Writer.valueClass(ParseText.class);
-org.apache.hadoop.io.SequenceFile.Writer.Option tProgressOpt =
SequenceFile.Writer.progressable(progress);
-org.apache.hadoop.io.SequenceFile.Writer.Option tCompOpt =
SequenceFile.Writer.compression(CompressionType.RECORD);
+final MapFile.Writer textOut;
+if (storeText) {
+ Option tKeyClassOpt = (Option) MapFile.Writer.keyClass(Text.class);
+ org.apache.hadoop.io.SequenceFile.Writer.Option tValClassOpt =
SequenceFile.Writer.valueClass(ParseText.class);
+ org.apache.hadoop.io.SequenceFile.Writer.Option tProgressOpt =
SequenceFile.Writer.progressable(progress);
+ org.apache.hadoop.io.SequenceFile.Writer.Option tCompOpt =
SequenceFile.Writer.compression(CompressionType.RECORD);
-final MapFile.Writer textOut = new MapFile.Writer(job, text,
+ textOut = new MapFile.Writer(job, text,
Review comment:
Format applied
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
> New configuration allowing to choose whether to store 'parse_text' directory
> or not.
>
>
> Key: NUTCH-2435
> URL: https://issues.apache.org/jira/browse/NUTCH-2435
> Project: Nutch
> Issue Type: New Feature
> Components: parser
>Affects Versions: 1.13
> Environment: Apach Nutch 1.13
>Reporter: Marcos Bori
>
> Whenever a page is parsed, one of the outputs is the directory 'parse_text'.
> It is intended to be used at the indexing phase so the page can be searched
> from a search engine such as Solr.
> In my special crawling case, I don't need to index the page contents.
> Therefore, creating and filing the 'parse_text' is not required for me. To
> optimize performance, I don't want the crawler to store this information to
> the filesystem.
> I propose a new parameter "parser.store.text" allowing to choose whether to
> store 'parse_text' directory or not. Its default value, of course, is "true".
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (NUTCH-2435) New configuration allowing to choose whether to store 'parse_text' directory or not.
[ https://issues.apache.org/jira/browse/NUTCH-2435?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16182602#comment-16182602 ] ASF GitHub Bot commented on NUTCH-2435: --- maborec commented on issue #225: NUTCH-2435 - New parameter "parser.store.text" URL: https://github.com/apache/nutch/pull/225#issuecomment-332528225 **eclipse-codeformat.xml** format applied to NUTCH-2435 changes. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org > New configuration allowing to choose whether to store 'parse_text' directory > or not. > > > Key: NUTCH-2435 > URL: https://issues.apache.org/jira/browse/NUTCH-2435 > Project: Nutch > Issue Type: New Feature > Components: parser >Affects Versions: 1.13 > Environment: Apach Nutch 1.13 >Reporter: Marcos Bori > > Whenever a page is parsed, one of the outputs is the directory 'parse_text'. > It is intended to be used at the indexing phase so the page can be searched > from a search engine such as Solr. > In my special crawling case, I don't need to index the page contents. > Therefore, creating and filing the 'parse_text' is not required for me. To > optimize performance, I don't want the crawler to store this information to > the filesystem. > I propose a new parameter "parser.store.text" allowing to choose whether to > store 'parse_text' directory or not. Its default value, of course, is "true". -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (NUTCH-2435) New configuration allowing to choose whether to store 'parse_text' directory or not.
[ https://issues.apache.org/jira/browse/NUTCH-2435?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16182445#comment-16182445 ] Sebastian Nagel commented on NUTCH-2435: +1 -- a valid use case (of course, not the most common one). PR looks good except formatting. > New configuration allowing to choose whether to store 'parse_text' directory > or not. > > > Key: NUTCH-2435 > URL: https://issues.apache.org/jira/browse/NUTCH-2435 > Project: Nutch > Issue Type: New Feature > Components: parser >Affects Versions: 1.13 > Environment: Apach Nutch 1.13 >Reporter: Marcos Bori > > Whenever a page is parsed, one of the outputs is the directory 'parse_text'. > It is intended to be used at the indexing phase so the page can be searched > from a search engine such as Solr. > In my special crawling case, I don't need to index the page contents. > Therefore, creating and filing the 'parse_text' is not required for me. To > optimize performance, I don't want the crawler to store this information to > the filesystem. > I propose a new parameter "parser.store.text" allowing to choose whether to > store 'parse_text' directory or not. Its default value, of course, is "true". -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (NUTCH-2435) New configuration allowing to choose whether to store 'parse_text' directory or not.
[
https://issues.apache.org/jira/browse/NUTCH-2435?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16182448#comment-16182448
]
ASF GitHub Bot commented on NUTCH-2435:
---
sebastian-nagel commented on a change in pull request #225: NUTCH-2435 - New
parameter "parser.store.text"
URL: https://github.com/apache/nutch/pull/225#discussion_r141321804
##
File path: src/java/org/apache/nutch/parse/ParseOutputFormat.java
##
@@ -128,13 +131,18 @@ public void checkOutputSpecs(FileSystem fs, JobConf job)
throws IOException {
.split(" *, *");
// textOut Options
-Option tKeyClassOpt = (Option) MapFile.Writer.keyClass(Text.class);
-org.apache.hadoop.io.SequenceFile.Writer.Option tValClassOpt =
SequenceFile.Writer.valueClass(ParseText.class);
-org.apache.hadoop.io.SequenceFile.Writer.Option tProgressOpt =
SequenceFile.Writer.progressable(progress);
-org.apache.hadoop.io.SequenceFile.Writer.Option tCompOpt =
SequenceFile.Writer.compression(CompressionType.RECORD);
+final MapFile.Writer textOut;
+if (storeText) {
+ Option tKeyClassOpt = (Option) MapFile.Writer.keyClass(Text.class);
+ org.apache.hadoop.io.SequenceFile.Writer.Option tValClassOpt =
SequenceFile.Writer.valueClass(ParseText.class);
+ org.apache.hadoop.io.SequenceFile.Writer.Option tProgressOpt =
SequenceFile.Writer.progressable(progress);
+ org.apache.hadoop.io.SequenceFile.Writer.Option tCompOpt =
SequenceFile.Writer.compression(CompressionType.RECORD);
-final MapFile.Writer textOut = new MapFile.Writer(job, text,
+ textOut = new MapFile.Writer(job, text,
tKeyClassOpt, tValClassOpt, tCompOpt, tProgressOpt);
+} else {
+ textOut=null;
Review comment:
see above
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
> New configuration allowing to choose whether to store 'parse_text' directory
> or not.
>
>
> Key: NUTCH-2435
> URL: https://issues.apache.org/jira/browse/NUTCH-2435
> Project: Nutch
> Issue Type: New Feature
> Components: parser
>Affects Versions: 1.13
> Environment: Apach Nutch 1.13
>Reporter: Marcos Bori
>
> Whenever a page is parsed, one of the outputs is the directory 'parse_text'.
> It is intended to be used at the indexing phase so the page can be searched
> from a search engine such as Solr.
> In my special crawling case, I don't need to index the page contents.
> Therefore, creating and filing the 'parse_text' is not required for me. To
> optimize performance, I don't want the crawler to store this information to
> the filesystem.
> I propose a new parameter "parser.store.text" allowing to choose whether to
> store 'parse_text' directory or not. Its default value, of course, is "true".
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (NUTCH-2435) New configuration allowing to choose whether to store 'parse_text' directory or not.
[
https://issues.apache.org/jira/browse/NUTCH-2435?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16182447#comment-16182447
]
ASF GitHub Bot commented on NUTCH-2435:
---
sebastian-nagel commented on a change in pull request #225: NUTCH-2435 - New
parameter "parser.store.text"
URL: https://github.com/apache/nutch/pull/225#discussion_r141321740
##
File path: src/java/org/apache/nutch/parse/ParseOutputFormat.java
##
@@ -128,13 +131,18 @@ public void checkOutputSpecs(FileSystem fs, JobConf job)
throws IOException {
.split(" *, *");
// textOut Options
-Option tKeyClassOpt = (Option) MapFile.Writer.keyClass(Text.class);
-org.apache.hadoop.io.SequenceFile.Writer.Option tValClassOpt =
SequenceFile.Writer.valueClass(ParseText.class);
-org.apache.hadoop.io.SequenceFile.Writer.Option tProgressOpt =
SequenceFile.Writer.progressable(progress);
-org.apache.hadoop.io.SequenceFile.Writer.Option tCompOpt =
SequenceFile.Writer.compression(CompressionType.RECORD);
+final MapFile.Writer textOut;
+if (storeText) {
+ Option tKeyClassOpt = (Option) MapFile.Writer.keyClass(Text.class);
+ org.apache.hadoop.io.SequenceFile.Writer.Option tValClassOpt =
SequenceFile.Writer.valueClass(ParseText.class);
+ org.apache.hadoop.io.SequenceFile.Writer.Option tProgressOpt =
SequenceFile.Writer.progressable(progress);
+ org.apache.hadoop.io.SequenceFile.Writer.Option tCompOpt =
SequenceFile.Writer.compression(CompressionType.RECORD);
-final MapFile.Writer textOut = new MapFile.Writer(job, text,
+ textOut = new MapFile.Writer(job, text,
Review comment:
Please, format using
[eclipse-codeformat.xml](https://github.com/apache/nutch/blob/master/eclipse-codeformat.xml).
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
> New configuration allowing to choose whether to store 'parse_text' directory
> or not.
>
>
> Key: NUTCH-2435
> URL: https://issues.apache.org/jira/browse/NUTCH-2435
> Project: Nutch
> Issue Type: New Feature
> Components: parser
>Affects Versions: 1.13
> Environment: Apach Nutch 1.13
>Reporter: Marcos Bori
>
> Whenever a page is parsed, one of the outputs is the directory 'parse_text'.
> It is intended to be used at the indexing phase so the page can be searched
> from a search engine such as Solr.
> In my special crawling case, I don't need to index the page contents.
> Therefore, creating and filing the 'parse_text' is not required for me. To
> optimize performance, I don't want the crawler to store this information to
> the filesystem.
> I propose a new parameter "parser.store.text" allowing to choose whether to
> store 'parse_text' directory or not. Its default value, of course, is "true".
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (NUTCH-2435) New configuration allowing to choose whether to store 'parse_text' directory or not.
[
https://issues.apache.org/jira/browse/NUTCH-2435?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16182449#comment-16182449
]
ASF GitHub Bot commented on NUTCH-2435:
---
sebastian-nagel commented on a change in pull request #225: NUTCH-2435 - New
parameter "parser.store.text"
URL: https://github.com/apache/nutch/pull/225#discussion_r141321863
##
File path: src/java/org/apache/nutch/parse/ParseOutputFormat.java
##
@@ -162,7 +170,9 @@ public void write(Text key, Parse parse) throws
IOException {
String fromUrl = key.toString();
// host or domain name of the source URL
String origin = null;
-textOut.append(key, new ParseText(parse.getText()));
+if (textOut!=null) {
Review comment:
see above, formatting
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
> New configuration allowing to choose whether to store 'parse_text' directory
> or not.
>
>
> Key: NUTCH-2435
> URL: https://issues.apache.org/jira/browse/NUTCH-2435
> Project: Nutch
> Issue Type: New Feature
> Components: parser
>Affects Versions: 1.13
> Environment: Apach Nutch 1.13
>Reporter: Marcos Bori
>
> Whenever a page is parsed, one of the outputs is the directory 'parse_text'.
> It is intended to be used at the indexing phase so the page can be searched
> from a search engine such as Solr.
> In my special crawling case, I don't need to index the page contents.
> Therefore, creating and filing the 'parse_text' is not required for me. To
> optimize performance, I don't want the crawler to store this information to
> the filesystem.
> I propose a new parameter "parser.store.text" allowing to choose whether to
> store 'parse_text' directory or not. Its default value, of course, is "true".
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (NUTCH-2435) New configuration allowing to choose whether to store 'parse_text' directory or not.
[ https://issues.apache.org/jira/browse/NUTCH-2435?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16182388#comment-16182388 ] ASF GitHub Bot commented on NUTCH-2435: --- maborec opened a new pull request #225: NUTCH-2435 - New parameter "parser.store.text" URL: https://github.com/apache/nutch/pull/225 NUTCH-2435 - New parameter "parser.store.text" allowing to choose whether to store 'parse_text' directory or not. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org > New configuration allowing to choose whether to store 'parse_text' directory > or not. > > > Key: NUTCH-2435 > URL: https://issues.apache.org/jira/browse/NUTCH-2435 > Project: Nutch > Issue Type: New Feature > Components: parser >Affects Versions: 1.13 > Environment: Apach Nutch 1.13 >Reporter: Marcos Bori > > Whenever a page is parsed, one of the outputs is the directory 'parse_text'. > It is intended to be used at the indexing phase so the page can be searched > from a search engine such as Solr. > In my special crawling case, I don't need to index the page contents. > Therefore, creating and filing the 'parse_text' is not required for me. To > optimize performance, I don't want the crawler to store this information to > the filesystem. > I propose a new parameter "parser.store.text" allowing to choose whether to > store 'parse_text' directory or not. Its default value, of course, is "true". -- This message was sent by Atlassian JIRA (v6.4.14#64029)
