[
https://issues.apache.org/jira/browse/OPENNLP-1148?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16215951#comment-16215951
]
ASF GitHub Bot commented on OPENNLP-1148:
-----------------------------------------
smarthi closed pull request #279: OPENNLP-1148: use StandardCharsets.UTF_8 for
the second parameter of …
URL: https://github.com/apache/opennlp/pull/279
This is a PR merged from a forked repository.
As GitHub hides the original diff on merge, it is displayed below for
the sake of provenance:
As this is a foreign pull request (from a fork), the diff is supplied
below (as it won't show otherwise due to GitHub magic):
diff --git a/opennlp-docs/src/docbkx/doccat.xml
b/opennlp-docs/src/docbkx/doccat.xml
index c056732eb..3c456b9a3 100644
--- a/opennlp-docs/src/docbkx/doccat.xml
+++ b/opennlp-docs/src/docbkx/doccat.xml
@@ -131,7 +131,7 @@ InputStream dataIn = null;
try (dataIn = new FileInputStream("en-sentiment.train")) {
ObjectStream<String> lineStream =
- new PlainTextByLineStream(dataIn, "UTF-8");
+ new PlainTextByLineStream(dataIn, StandardCharsets.UTF_8);
ObjectStream<DocumentSample> sampleStream = new
DocumentSampleStream(lineStream);
model = DocumentCategorizerME.train("en", sampleStream);
@@ -153,4 +153,4 @@ try (OutputStream modelOut = new BufferedOutputStream(new
FileOutputStream(model
</para>
</section>
</section>
-</chapter>
\ No newline at end of file
+</chapter>
diff --git a/opennlp-docs/src/docbkx/langdetect.xml
b/opennlp-docs/src/docbkx/langdetect.xml
index 67412a4c6..a5ca6087c 100644
--- a/opennlp-docs/src/docbkx/langdetect.xml
+++ b/opennlp-docs/src/docbkx/langdetect.xml
@@ -205,7 +205,7 @@ $ tail -20000 < leipzig_shuf.txt > leipzig.eval
InputStreamFactory inputStreamFactory = new MarkableFileInputStreamFactory(new
File("corpus.txt"));
ObjectStream<String> lineStream =
- new PlainTextByLineStream(inputStreamFactory, "UTF-8");
+ new PlainTextByLineStream(inputStreamFactory, StandardCharsets.UTF_8);
ObjectStream<LanguageSample> sampleStream = new
LanguageDetectorSampleStream(lineStream);
TrainingParameters params = ModelUtil.createDefaultTrainingParameters();
@@ -223,4 +223,4 @@ model.serialize(new File("langdetect.bin"));
</para>
</section>
</section>
-</chapter>
\ No newline at end of file
+</chapter>
diff --git a/opennlp-docs/src/docbkx/lemmatizer.xml
b/opennlp-docs/src/docbkx/lemmatizer.xml
index 630b04d4e..60fba8f72 100644
--- a/opennlp-docs/src/docbkx/lemmatizer.xml
+++ b/opennlp-docs/src/docbkx/lemmatizer.xml
@@ -302,7 +302,7 @@ InputStreamFactory inputStreamFactory = null;
LemmaSampleStream lemmaStream = null;
try {
lineStream = new PlainTextByLineStream(
- (inputStreamFactory), "UTF-8");
+ (inputStreamFactory), StandardCharsets.UTF_8);
lemmaStream = new LemmaSampleStream(lineStream);
} catch (IOException e) {
CmdLineUtil.handleCreateObjectStreamError(e);
@@ -357,4 +357,4 @@ Accuracy: 0.9659110277825124]]>
</screen>
</para>
</section>
-</chapter>
\ No newline at end of file
+</chapter>
diff --git a/opennlp-docs/src/docbkx/namefinder.xml
b/opennlp-docs/src/docbkx/namefinder.xml
index abcd09430..a5735b0c8 100644
--- a/opennlp-docs/src/docbkx/namefinder.xml
+++ b/opennlp-docs/src/docbkx/namefinder.xml
@@ -268,7 +268,7 @@ $ opennlp TokenNameFinderTrainer -featuregen brown.xml
-sequenceCodec BILOU -res
<programlisting language="java">
<![CDATA[
ObjectStream<String> lineStream =
- new PlainTextByLineStream(new
FileInputStream("en-ner-person.train"), StandardCharsets.UTF8);
+ new PlainTextByLineStream(new
FileInputStream("en-ner-person.train"), StandardCharsets.UTF_8);
TokenNameFinderModel model;
diff --git a/opennlp-docs/src/docbkx/parser.xml
b/opennlp-docs/src/docbkx/parser.xml
index 614293b96..12de5fb61 100644
--- a/opennlp-docs/src/docbkx/parser.xml
+++ b/opennlp-docs/src/docbkx/parser.xml
@@ -266,7 +266,7 @@ CmdLineUtil.checkOutputFile("parser model", modelOutFile);
try {
HeadRules rules = createHeadRules(params);
InputStreamFactory inputStreamFactory = new
MarkableFileInputStreamFactory(new File("parsing.train"));
- ObjectStream<String> stringStream = new
PlainTextByLineStream(inputStreamFactory, "UTF-8");
+ ObjectStream<String> stringStream = new
PlainTextByLineStream(inputStreamFactory, StandardCharsets.UTF_8);
ObjectStream<Parse> sampleStream = new ParseSample(stringStream);
ParserType type = parseParserType(params.getParserType());
@@ -366,7 +366,7 @@ System.out.println(result.toString());]]>
<programlisting language="java">
<![CDATA[
InputStreamFactory inputStreamFactory = new MarkableFileInputStreamFactory(new
File("parsing.train"));
-ObjectStream<String> stringStream = new
PlainTextByLineStream(inputStreamFactory, "UTF-8");
+ObjectStream<String> stringStream = new
PlainTextByLineStream(inputStreamFactory, StandardCharsets.UTF_8);
ObjectStream<Parse> sampleStream = new ParseSample(stringStream);
ParserCrossValidator evaluator = new ParserCrossValidator("en",
trainParameters, headRules, \
parserType, listeners.toArray(new ParserEvaluationMonitor[listeners.size()])));
diff --git a/opennlp-docs/src/docbkx/tokenizer.xml
b/opennlp-docs/src/docbkx/tokenizer.xml
index 6d54c3c40..3fb451935 100644
--- a/opennlp-docs/src/docbkx/tokenizer.xml
+++ b/opennlp-docs/src/docbkx/tokenizer.xml
@@ -311,9 +311,8 @@ Path: en-token.bin]]>
The following sample code illustrates these steps:
<programlisting language="java">
<![CDATA[
-Charset charset = Charset.forName("UTF-8");
ObjectStream<String> lineStream = new PlainTextByLineStream(new
FileInputStream("en-sent.train"),
- charset);
+ StandardCharsets.UTF_8);
ObjectStream<TokenSample> sampleStream = new TokenSampleStream(lineStream);
TokenizerModel model;
@@ -411,4 +410,4 @@ are very welcome. If you want to contribute please contact
us on the mailing lis
or comment on the jira issue <ulink
url="https://issues.apache.org/jira/browse/OPENNLP-217";>OPENNLP-217</ulink>.</para>
</section>
</section>
-</chapter>
\ No newline at end of file
+</chapter>
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
[email protected]
> use StandardCharsets.UTF_8 in doc
> ---------------------------------
>
> Key: OPENNLP-1148
> URL: https://issues.apache.org/jira/browse/OPENNLP-1148
> Project: OpenNLP
> Issue Type: Improvement
> Components: Documentation
> Affects Versions: 1.8.2
> Reporter: Koji Sekiguchi
> Assignee: Koji Sekiguchi
> Priority: Trivial
> Fix For: 1.8.3
>
>
> In the doc, the use of PlainTextByLineStream() is not unified. Other than
> specifying StandardCharsets.UTF_8 for its second parameter, there are
> following variations:
> - String "UTF-8"
> - StandardCharsets.UTF8 (not UTF_8)
> - Charset.forName("UTF-8")
> Let's unify the use to StandardCharsets.UTF_8
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
