krickert commented on code in PR #1457:
URL: https://github.com/apache/opennlp/pull/1457#discussion_r4235528223
##########
opennlp-docs/src/docbkx/tokenizer.xml:
##########
@@ -240,11 +240,10 @@ boolean word = policy.test("Cafe\u0301");]]>
Use <code>WhitespaceTokenizer.newInstance(boolean
keepNewLines)</code> when a
caller needs its own whitespace tokenizer
configuration. Each call returns a
distinct instance. Changing it does not change
- <code>WhitespaceTokenizer.INSTANCE</code>. Since 3.0.0,
- <code>TokenizerME</code> uses private whitespace
tokenizers and no longer changes
- the shared instance when applying its
<code>keepNewLines</code> setting.
- The shared instance remains mutable; callers needing
different settings should
- use separate instances rather than reconfigure the
singleton.
+ <code>WhitespaceTokenizer.INSTANCE</code>.
<code>TokenizerME</code> uses its own
+ whitespace tokenizers and does not change the shared
instance. The shared
+ instance is mutable; callers that need different
settings should use separate
+ instances instead of reconfiguring it.
Review Comment:
I really like how much easier it flows. Nice changes..
##########
opennlp-docs/src/docbkx/langdetect.xml:
##########
@@ -58,16 +58,17 @@ under the License.
<row>
<entry>UrlCharSequenceNormalizer</entry>
<entry>Replaces each
http or https URL and each email address with a
- blank space.
Since 3.0.0 a URL is removed as a whole, up to the next
+ blank space. A
URL is removed as a whole, up to the next
whitespace or
delimiter, including a port, user information, percent
escapes and
non-ASCII hosts; URLs with other schemes are kept. Models
- trained with an
earlier release saw only the leading ASCII part of a
+ trained with a
release before 3.0.0 saw only the leading ASCII part of a
Review Comment:
good catch, versionless mention is confusing
##########
opennlp-docs/src/docbkx/tokenizer.xml:
##########
@@ -640,6 +614,31 @@ UnigramSegmenter segmenter =
UnigramSegmenter.load(Path.of("chinese-words.txt"))
String[] words = segmenter.tokenize("我来到北京清华大学");]]>
</programlisting>
</para>
+ <para>
+ <code>matrix.def</code> must list a cost for
every declared cell; matrix
+ dimensions plus lexicon size are bounded by the
shared
+ <code>ResourceLimits.MAX_ENTRIES</code> limit
and the matrix cell count
+ by <code>ResourceLimits.MAX_MATRIX_CELLS</code>.
+ Dictionary loading rejects empty lexicon
surfaces, duplicate
+ <code>matrix.def</code> entries, duplicate
<code>char.def</code>
+ categories, costs outside the signed 16-bit
range, malformed text in the
+ selected encoding, and <code>unk.def</code>
templates for undefined
+ categories. MeCab-quoted CSV fields are
accepted, with three
+ deviations from MeCab's reader: quotes open and
close a quoted stretch
+ wherever they appear in a field, text after a
closing quote joins the
+ field instead of being dropped, and an
unterminated quoted field is
+ rejected with its file and line instead of
being read silently to the
+ end of the line.
+ </para>
+ <para>
+ A <code>char.def</code> may define up to 17
categories, and category
+ lengths range from 0 through 15, matching the
MeCab character-property
+ compiler.
+ If a <code>char.def</code> mapping lists
multiple categories, the first
+ provides unknown-word settings. Grouping
continues while successive
+ assignments overlap. Unknown words use their
category's <code>unk.def</code>
+ template first, with <code>DEFAULT</code> used
when no category template exists.
Review Comment:
This looks better. Spent a lot of time thinking about the deviations, it
was too much. Glad you shortened it
##########
opennlp-docs/src/docbkx/introduction.xml:
##########
@@ -318,18 +318,17 @@ Arguments description:
<section xml:id="intro.stream.resources">
<title>Closing and resetting input streams</title>
<para>
- Close corpus streams after use, preferably with
try-with-resources. Since 3.0.0,
+ Close corpus streams after use, preferably with
try-with-resources.
<code>PlainTextByLineStream.reset()</code> closes the
reader from the previous
pass before asking its <code>InputStreamFactory</code>
for a new stream. The
- factory must therefore provide a fresh, readable stream
for each pass, rather
+ factory must therefore provide a new, readable stream
for each pass, rather
than return the stream that has just been closed.
</para>
<para>
The markable file stream that the command line tools
and the format converters open
for corpus files releases its file handle on close,
including when closed through
a wrapping reader. The Leipzig language-sample reader
also
- closes the stream used to count source lines. These
fixes prevent file handles
- from accumulating during repeated reads and training
passes.
+ closes the stream used to count source lines.
Review Comment:
agree on sentence removal, too specific
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
