This is an automated email from the ASF dual-hosted git repository.
mawiesne pushed a commit to branch main
in repository https://gitbox.apache.org/repos/asf/opennlp.git
The following commit(s) were added to refs/heads/main by this push:
new 46d9d96a1 GH-1456: Tighten normalizer, tokenizer, introduction and
language detector chapters (#1457)
46d9d96a1 is described below
commit 46d9d96a10570bdd69523c92303e00936a49eab4
Author: Richard Zowalla <[email protected]>
AuthorDate: Sat Oct 10 11:07:54 2026 +0200
GH-1456: Tighten normalizer, tokenizer, introduction and language detector
chapters (#1457)
---
opennlp-docs/src/docbkx/introduction.xml | 47 ++++++++-----------
opennlp-docs/src/docbkx/langdetect.xml | 13 +++---
opennlp-docs/src/docbkx/namefinder.xml | 2 +-
opennlp-docs/src/docbkx/normalizer.xml | 80 +++++++++++++++++---------------
opennlp-docs/src/docbkx/tokenizer.xml | 63 +++++++++++++------------
5 files changed, 100 insertions(+), 105 deletions(-)
diff --git a/opennlp-docs/src/docbkx/introduction.xml
b/opennlp-docs/src/docbkx/introduction.xml
index 93dc7f286..109679a95 100644
--- a/opennlp-docs/src/docbkx/introduction.xml
+++ b/opennlp-docs/src/docbkx/introduction.xml
@@ -318,18 +318,17 @@ Arguments description:
<section xml:id="intro.stream.resources">
<title>Closing and resetting input streams</title>
<para>
- Close corpus streams after use, preferably with
try-with-resources. Since 3.0.0,
+ Close corpus streams after use, preferably with
try-with-resources.
<code>PlainTextByLineStream.reset()</code> closes the
reader from the previous
pass before asking its <code>InputStreamFactory</code>
for a new stream. The
- factory must therefore provide a fresh, readable stream
for each pass, rather
+ factory must therefore provide a new, readable stream
for each pass, rather
than return the stream that has just been closed.
</para>
<para>
The markable file stream that the command line tools
and the format converters open
for corpus files releases its file handle on close,
including when closed through
a wrapping reader. The Leipzig language-sample reader
also
- closes the stream used to count source lines. These
fixes prevent file handles
- from accumulating during repeated reads and training
passes.
+ closes the stream used to count source lines.
</para>
</section>
<section xml:id="intro.sysprops">
@@ -355,38 +354,30 @@ Arguments description:
<section xml:id="intro.sysprops.whitespace">
<title>Whitespace Definition</title>
<para>
- What counts as whitespace is a common interoperability gap
across programming
- languages and libraries. The Unicode Character Database
defines a binary
- <code>White_Space</code> property. As documented on
- <link
xlink:href="https://en.wikipedia.org/wiki/Whitespace_character";>Wikipedia's
- whitespace character page</link>, Unicode defines exactly
twenty-five characters
- with <code>White_Space=yes</code> (<quote>WSpace=Y</quote> /
<quote>WS</quote>),
- including ordinary space and tab, the next-line control
(<code>U+0085</code>),
- no-break and typographic spaces such as <code>U+00A0</code> and
- <code>U+2007</code>, and the line and paragraph separators
- (<code>U+2028</code>/<code>U+2029</code>). That page also
states the cross-language
- weakness directly: <quote>Most languages only recognize
whitespace characters that
- have an ASCII code. They disallow most or all of the Unicode
codes listed
- above</quote>. Java is a concrete example of that mismatch:
+ The Unicode standard defines whitespace as the 25 code points
with the binary
+ <code>White_Space</code> property
+ (<link
xlink:href="https://www.unicode.org/Public/17.0.0/ucd/PropList.txt";><code>PropList.txt</code></link>,
+ Unicode 17.0.0). The set includes space and tab, the next-line
control
+ (<code>U+0085</code>), no-break and typographic spaces such as
<code>U+00A0</code>
+ and <code>U+2007</code>, and the line and paragraph separators
+ (<code>U+2028</code>/<code>U+2029</code>). Java's
<code>Character.isWhitespace</code> omits the no-break spaces
and
- <code>U+0085</code> while including the information separators
- <code>U+001C</code>..<code>U+001F</code>, and Java's default
<code>\s</code> regex
- class is likewise an ASCII-oriented subset unless Unicode
character-class mode is
- enabled. Text extracted from PDFs, HTML, and word processors
regularly carries the
- wider set, so a tokenizer that follows the JVM predicate and
one that follows
- Unicode will disagree on those documents.
+ <code>U+0085</code> but includes the information separators
+ <code>U+001C</code>..<code>U+001F</code>; Java's default
<code>\s</code> regex
+ class matches ASCII whitespace only unless Unicode
character-class mode is
+ enabled. Text extracted from PDFs, HTML, and word processors
often contains the
+ wider set, on which the two definitions produce different
tokens.
</para>
<para>
Tokenization, corpus format parsing, and feature generation
therefore classify
whitespace using the Unicode <code>White_Space</code> property
by default, via
<code>StringUtil.isWhitespace</code> and the related
<code>StringUtil.splitOnUnicodeWhitespace</code> /
- <code>trimUnicodeWhitespace</code> helpers (code-point scans,
not regular
- expressions). Models trained with OpenNLP 1.x or 2.x used the
legacy definition
+ <code>trimUnicodeWhitespace</code> helpers. Models trained
with OpenNLP 1.x or
+ 2.x, which include the models published by the project, used
the legacy definition
(the union of <code>Character.isWhitespace</code> and the
Unicode <code>Zs</code>
- category). As of this release, that covers all published
OpenNLP models, so set
- the property below when decoding those models if you need
byte-identical
- tokenization at the few code points where the two definitions
differ.
+ category). Set the property below when applying those models
to reproduce their
+ tokenization at the code points where the two definitions
differ.
</para>
<para>
Users may switch to the legacy definition by setting the
following system
diff --git a/opennlp-docs/src/docbkx/langdetect.xml
b/opennlp-docs/src/docbkx/langdetect.xml
index dbc295a33..cb78406e4 100644
--- a/opennlp-docs/src/docbkx/langdetect.xml
+++ b/opennlp-docs/src/docbkx/langdetect.xml
@@ -58,16 +58,17 @@ under the License.
<row>
<entry>UrlCharSequenceNormalizer</entry>
<entry>Replaces each
http or https URL and each email address with a
- blank space.
Since 3.0.0 a URL is removed as a whole, up to the next
+ blank space. A
URL is removed as a whole, up to the next
whitespace or
delimiter, including a port, user information, percent
escapes and
non-ASCII hosts; URLs with other schemes are kept. Models
- trained with an
earlier release saw only the leading ASCII part of a
+ trained with a
release before 3.0.0 saw only the leading ASCII part of a
URL removed and
get that output again under
<code>-Dopennlp.compat.mode=LEGACY</code> until they are retrained.</entry>
</row>
<row>
-
<entry>TwitterCharSequenceNormalizer</entry>
- <entry>Replaces
hashtags and Twitter usernames by blank spaces.</entry>
+
<entry>SocialMediaCharSequenceNormalizer</entry>
+ <entry>Replaces each
hashtag, user handle, run of retweet markers and emoticon
+ with one space,
and shrinks repeated laughter to two repetitions.</entry>
</row>
<row>
<entry>NumberCharSequenceNormalizer</entry>
@@ -82,8 +83,8 @@ under the License.
</table>
</para>
<para>
- Since 3.0.0, EmojiCharSequenceNormalizer removes
complete emoji only. Models trained with an earlier
- release saw every hyphen, every character from U+E000
to U+FFFF and nearly every supplementary
+ EmojiCharSequenceNormalizer removes complete emoji
only. Models trained with a release before
+ 3.0.0 saw every hyphen, every character from U+E000 to
U+FFFF and nearly every supplementary
character removed, emoji or not. They get that output
again under
<code>-Dopennlp.compat.mode=LEGACY</code> until they
are retrained.
</para>
diff --git a/opennlp-docs/src/docbkx/namefinder.xml
b/opennlp-docs/src/docbkx/namefinder.xml
index 43bbeb58a..359dec9a3 100644
--- a/opennlp-docs/src/docbkx/namefinder.xml
+++ b/opennlp-docs/src/docbkx/namefinder.xml
@@ -185,7 +185,7 @@ Span[] spans = nameFinderDL.findInOriginal(tokens);]]>
<code>White_Space</code> set before
WordPiece tokenization, so spacing such as a
no-break space or the CJK ideographic
space is recognized as a delimiter. After
inference, reconstructed entity text is
matched back to the caller's original input
- with a Unicode-aware cursor scan (not a
regular expression), so
+ with a Unicode-aware cursor scan, so
<code>Span#getCoveredText(...)</code>
returns the source text even when WordPiece
rejoins sub-tokens with spaces or when
the source uses non-ASCII whitespace between
tokens.
diff --git a/opennlp-docs/src/docbkx/normalizer.xml
b/opennlp-docs/src/docbkx/normalizer.xml
index 7950772ee..47661ecd8 100644
--- a/opennlp-docs/src/docbkx/normalizer.xml
+++ b/opennlp-docs/src/docbkx/normalizer.xml
@@ -35,7 +35,9 @@
<para>
<emphasis
role="bold">Standards-sourced.</emphasis> Membership sets come from the
Unicode Character Database (for example
the <code>White_Space</code> and
- <code>Dash</code> properties), not from
the JVM's locale-dependent or quirky
+ <code>Dash</code> properties in
+ <link
xlink:href="https://www.unicode.org/Public/17.0.0/ucd/PropList.txt";><code>PropList.txt</code></link>,
+ Unicode 17.0.0), not from the JVM's
locale-dependent or quirky
character predicates. The normalization
engine never relies on
<code>Character.isWhitespace</code>,
which disagrees with the Unicode standard.
</para>
@@ -158,7 +160,15 @@
<row>
<entry><code>CaseFoldCharSequenceNormalizer</code></entry>
<entry>Lower cases for
case-insensitive matching, using
-
<code>Locale.ROOT</code>.</entry>
+
<code>Locale.ROOT</code> or a supplied locale. Does not apply full case
+ folding; U+00DF is
unchanged.</entry>
+ </row>
+ <row>
+
<entry><code>FullCaseFoldCharSequenceNormalizer</code></entry>
+ <entry>Applies Unicode full
case folding (<code>CaseFolding.txt</code>,
+ status <code>C</code>
and <code>F</code>), including the expanding
+ folds such as U+00DF to
<code>ss</code> and U+FB00 to <code>ff</code>.
+ The Turkic mappings are
not applied. Expects NFC input.</entry>
</row>
<row>
<entry><code>AccentFoldCharSequenceNormalizer</code></entry>
@@ -298,9 +308,8 @@ Span hit = aligned.toOriginalSpan(5, 14); // "the-match"
in the normalized tex
<code>NameFinderDL</code> additionally locates
reconstructed entity text in the original
input with a cursor-based matcher: a space in the
reconstructed span matches zero or more
Unicode whitespace code points in the source, and every
other code point is compared
- case-insensitively. This replaces the previous
regular-expression approach and correctly
- handles spacing copied from PDFs, the web, or non-Latin
sources when resolving
- <code>Span#getCoveredText(...)</code>.
+ case-insensitively. This handles spacing copied from
PDFs, the web, or non-Latin sources
+ when resolving <code>Span#getCoveredText(...)</code>.
</para>
<para>
Optional input folding is controlled through
<code>InferenceOptions</code> and is
@@ -403,7 +412,7 @@ String[] tokens = ws.split("one two"); // ["one", "two"]
(offset-aware via spl
dash.normalize("a—b"); // "a-b"]]>
</programlisting>
<para>
- A class applies its replacement three ways, which
differ in whether they collapse runs and
+ A class applies its replacement in four ways, which
differ in whether they collapse runs and
whether they preserve character offsets:
</para>
<itemizedlist>
@@ -514,20 +523,20 @@ Term term = analyzer.analyze("Running").get(0);
unrequested dimension costs one transform. The
character-level dimensions have built-in
defaults; <code>STEM</code> and <code>LEMMA</code>
require a
<code>Stemmer</code> or <code>Lemmatizer</code> (and
<code>LEMMA</code> a part-of-speech
- tag), and fail loudly if requested without them. An
analyzer is immutable and safe to share
- across threads when its configured transforms are. The
built-in character normalizers are
+ tag); requesting either without them throws an
<code>IllegalStateException</code>.
+ An analyzer is immutable and safe to share across
threads when its configured transforms are. The built-in character normalizers
are
stateless and <code>SnowballStemmer</code> is
thread-safe, so they can be shared as is. A
<code>Stemmer</code> passed to
<code>stem(Stemmer)</code> is used directly, so a stateful one
such as <code>PorterStemmer</code> confines the
analyzer to one thread. Passing a
<code>StemmerFactory</code> to
<code>stem(StemmerFactory)</code> instead wraps it in a
<code>CachingStemmer</code> (see <xref
linkend="tools.stemmer.sharing"/>), which gives each
- thread its own stemmer and cache, so the analyzer is
shareable even when the factory mints
+ thread its own stemmer and cache, so the analyzer is
shareable even when the factory creates
stateful stemmers.
</para>
<para>
Each dimension's transform is configurable on the
builder. Beyond the no-argument methods
- that enable a dimension with its default, there are
convenience methods for the common
- knobs, and a general <code>transform(dimension,
normalizer)</code> escape hatch for any
+ that enable a dimension with its default, there are
convenience methods for common
+ options, and the general method
<code>transform(dimension, normalizer)</code> for any
character-level dimension:
</para>
<programlisting language="java">
@@ -551,18 +560,15 @@ Term term = analyzer.analyze("Running").get(0);
<section xml:id="tools.normalizer.emojiannotations">
<title>Emoji annotations</title>
<para>
- Beyond folding, a pictograph carries signal worth
keeping: a name, a coarse sentiment, an
- entity type, a document category, and for flags a
region. <code>EmojiAnnotator</code> is the
- opt-in surface for that signal. It assembles one
<code>EmojiAnnotation</code> record per
- symbol from three strictly separated layers:
<emphasis>bundled facts</emphasis> from the
- provenance-tagged, project-authored
<code>emoji-annotations.txt</code> table,
- <emphasis>derived facts</emphasis> computed from the
code point sequence itself
- (<code>EmojiFlags</code> decodes a regional indicator
pair to its ISO 3166-1 alpha-2 code and
- a subdivision tag sequence to its ISO 3166-2 code, with
no data file), and optional
- <emphasis>joined facts</emphasis> resolved at run time
through an
- <code>EmojiAnnotationJoin</code> hook against
user-installed data such as a gazetteer.
- Every attribute value carries its own provenance
source, and gazetteer identifiers are never
- baked into the bundled table.
+ <code>EmojiAnnotator</code> provides optional metadata
for a pictograph: a name, a coarse
+ sentiment, an entity type, a document category and, for
flags, a region. It assembles one
+ <code>EmojiAnnotation</code> record per symbol from
three separate sources: the bundled,
+ project-authored <code>emoji-annotations.txt</code>
table; values computed from the code
+ point sequence itself (<code>EmojiFlags</code> decodes
a regional indicator pair to its
+ ISO 3166-1 alpha-2 code and a subdivision tag sequence
to its ISO 3166-2 code, with no data
+ file); and optional values resolved at run time through
an <code>EmojiAnnotationJoin</code>
+ hook against user-installed data such as a gazetteer.
Every attribute value records its
+ source. The bundled table contains no gazetteer
identifiers.
</para>
<programlisting language="java">
<![CDATA[EmojiAnnotator annotator = new EmojiAnnotator();
@@ -581,31 +587,29 @@ for (Term term : analyzer.analyze(text)) {
}]]>
</programlisting>
<para>
- <code>EmojiAnnotatorUsageExampleTest</code> asserts the
behavior shown here.
- Annotations are per-symbol
<emphasis>metadata</emphasis>, not text transforms, so they are a
- parallel accessor surface beside <code>Term</code>
rather than additional
- <code>Dimension</code>s: <code>annotate(Term)</code>
reads the token's original layer, which
- is the symbol the author actually wrote even after a
fold dimension has rewritten it to
- ASCII. Annotation is total over arbitrary tokens (a
non-emoji token is simply empty, and
- degenerate flag-shaped text yields no region instead of
an exception), while the strict
- decoder <code>EmojiFlags.isoRegion()</code> fails
loudly on malformed sequences for callers
- that want the contract. The bundled table's sentiment
scores are original project judgments
+ Annotations are per-symbol
<emphasis>metadata</emphasis>, not text transforms, and are
+ therefore not <code>Dimension</code>s:
<code>annotate(Term)</code> reads the token's original
+ layer, the symbol as written, even after a fold
dimension has rewritten it. It accepts any
+ token: a non-emoji token yields an empty
+ result, and malformed flag-shaped text yields no
region. The strict decoder
+ <code>EmojiFlags.isoRegion()</code> throws an
<code>IllegalArgumentException</code> on
+ malformed flag sequences. The bundled table's sentiment
scores are original project judgments
(tagged <code>UNSPECIFIED</code>); its names and its
entity-type/category assignments derive
from the Unicode-licensed CLDR short names and
<code>emoji-test.txt</code> group headers, and
no third-party sentiment data set is included.
</para>
<para>
- The annotations are wired as strictly opt-in feature
inputs to three components, each
- through its existing extension seam so that default
behavior and existing models are
- unchanged: the name finder (and any consumer of feature
generation descriptors) activates
+ The annotations are optional feature inputs to three
components, each added through an
+ existing extension point, so default behavior and
existing models are unchanged: the name
+ finder (and any consumer of feature generation
descriptors) activates
<code>EmojiAnnotationFeatureGenerator</code> by naming
<code>EmojiAnnotationFeatureGeneratorFactory</code> in
a descriptor, the document
categorizer takes <code>EmojiFeatureGenerator</code>
alongside its defaults through the
<code>DoccatFactory</code> feature generator array, and
sentiment analysis trains with
<code>EmojiSentimentFactory</code>, whose context
generator adds the emoji evidence and is
- re-created from the model manifest at prediction time.
A flag token then carries
- gazetteer-like evidence (<code>emojiType=FLAG</code>,
<code>emojiRegion=DE</code>) with no
- dictionary, and a heart or crying face contributes its
coarse polarity directly.
+ re-created from the model manifest at prediction time.
A flag token then yields the
+ features <code>emojiType=FLAG</code> and
<code>emojiRegion=DE</code> without a dictionary,
+ and a heart or crying face contributes its coarse
polarity directly.
</para>
</section>
diff --git a/opennlp-docs/src/docbkx/tokenizer.xml
b/opennlp-docs/src/docbkx/tokenizer.xml
index 34a702429..2eb69fad2 100644
--- a/opennlp-docs/src/docbkx/tokenizer.xml
+++ b/opennlp-docs/src/docbkx/tokenizer.xml
@@ -240,11 +240,10 @@ boolean word = policy.test("Cafe\u0301");]]>
Use <code>WhitespaceTokenizer.newInstance(boolean
keepNewLines)</code> when a
caller needs its own whitespace tokenizer
configuration. Each call returns a
distinct instance. Changing it does not change
- <code>WhitespaceTokenizer.INSTANCE</code>. Since 3.0.0,
- <code>TokenizerME</code> uses private whitespace
tokenizers and no longer changes
- the shared instance when applying its
<code>keepNewLines</code> setting.
- The shared instance remains mutable; callers needing
different settings should
- use separate instances rather than reconfigure the
singleton.
+ <code>WhitespaceTokenizer.INSTANCE</code>.
<code>TokenizerME</code> uses its own
+ whitespace tokenizers and does not change the shared
instance. The shared
+ instance is mutable; callers that need different
settings should use separate
+ instances instead of reconfiguring it.
</para>
</section>
<section xml:id="tools.tokenizer.training">
@@ -525,8 +524,7 @@ DetokenizationDictionary dict = new
DetokenizationDictionary(tokens, operations)
<![CDATA[int[] boundaries = WordSegmenter.boundaries("The quick brown fox.");
List<Span> segments = WordSegmenter.segments("The quick brown fox.");]]>
</programlisting>
- For allocation-free processing of large inputs,
stream the segments to a callback instead
- of collecting them.
+ To process the segments without building a
result list, stream them to a callback.
<programlisting language="java">
<![CDATA[WordSegmenter.forEachSegment("The quick brown fox.", (start, end) -> {
// handle the segment [start, end)
@@ -564,8 +562,7 @@ for (WordToken token : wordTokenizer.tokenizeTyped("OpenNLP
3.0")) {
<code>HANGUL</code>,
<code>SOUTHEAST_ASIAN</code>, and <code>EMOJI</code>.
</para>
<para>
- A streaming overload reports each token to a
handler with no per-token allocation, which
- is the fastest option when the tokens are
consumed on the fly.
+ A streaming overload reports each token to a
handler without building a result list.
<programlisting language="java">
<![CDATA[WordTokenizer wordTokenizer = new WordTokenizer();
wordTokenizer.tokenize("The quick brown fox.", (start, end, type) -> {
@@ -593,10 +590,6 @@ wordTokenizer.tokenize("The quick brown fox.", (start,
end, type) -> {
expected SHA-256 or SHA-512 digest; an
application-supplied opt-in catalog
(<code>installFromCatalog</code>) also needs
<code>-Dopennlp.download.remote=true</code>.
- <code>matrix.def</code> must list a cost for
every declared cell; matrix
- dimensions plus lexicon size are bounded by the
shared
- <code>ResourceLimits.MAX_ENTRIES</code> limit
and the matrix cell count
- by <code>ResourceLimits.MAX_MATRIX_CELLS</code>.
Fetching, verification, and unpacking use
<code>ResourceInstaller</code>,
described in the model loading chapter. Default
limits are a 1 GiB
download, 4 GiB unpacked, and 100000 archive
entries. Set higher limits at
@@ -607,25 +600,6 @@ wordTokenizer.tokenize("The quick brown fox.", (start,
end, type) -> {
<code>opennlp.install.max.expansion.ratio</code> for larger dictionaries such as
UniDic. Installed files are flattened to their
base names. Existing target
files are not replaced and must be removed
before a refresh.
- Dictionary loading rejects empty
lexicon surfaces, duplicate
- <code>matrix.def</code> entries,
duplicate <code>char.def</code>
- categories, costs outside the signed
16-bit range, malformed text in the
- selected encoding, and
<code>unk.def</code> templates for undefined
- categories. MeCab-quoted CSV fields are
accepted, with three documented
- deviations from MeCab's reader: quotes
open and close a quoted stretch
- wherever they appear in a field, text
after a closing quote joins the
- field instead of being dropped, and an
unterminated quoted field is
- rejected with its file and line instead
of being read silently to the
- end of the line.
- A <code>char.def</code> may define up
to 17 categories, and category
- lengths range from 0 through 15,
matching the MeCab character-property
- compiler.
- If a <code>char.def</code> mapping
lists multiple categories, the first
- provides unknown-word settings.
Grouping continues while successive
- assignments overlap. Unknown words use
their category's <code>unk.def</code>
- template first, with
<code>DEFAULT</code> used when no category template exists.
- <code>LatticeUsageExampleTest</code> asserts
the install-load-tokenize and
- lexicon flows shown here.
<programlisting language="java">
<![CDATA[MecabDictionaryInstaller.install(archiveUri, dictionaryDirectory,
sha512);
LatticeTokenizer tokenizer =
@@ -640,6 +614,31 @@ UnigramSegmenter segmenter =
UnigramSegmenter.load(Path.of("chinese-words.txt"))
String[] words = segmenter.tokenize("我来到北京清华大学");]]>
</programlisting>
</para>
+ <para>
+ <code>matrix.def</code> must list a cost for
every declared cell; matrix
+ dimensions plus lexicon size are bounded by the
shared
+ <code>ResourceLimits.MAX_ENTRIES</code> limit
and the matrix cell count
+ by <code>ResourceLimits.MAX_MATRIX_CELLS</code>.
+ Dictionary loading rejects empty lexicon
surfaces, duplicate
+ <code>matrix.def</code> entries, duplicate
<code>char.def</code>
+ categories, costs outside the signed 16-bit
range, malformed text in the
+ selected encoding, and <code>unk.def</code>
templates for undefined
+ categories. MeCab-quoted CSV fields are
accepted, with three
+ deviations from MeCab's reader: quotes open and
close a quoted stretch
+ wherever they appear in a field, text after a
closing quote joins the
+ field instead of being dropped, and an
unterminated quoted field is
+ rejected with its file and line instead of
being read silently to the
+ end of the line.
+ </para>
+ <para>
+ A <code>char.def</code> may define up to 17
categories, and category
+ lengths range from 0 through 15, matching the
MeCab character-property
+ compiler.
+ If a <code>char.def</code> mapping lists
multiple categories, the first
+ provides unknown-word settings. Grouping
continues while successive
+ assignments overlap. Unknown words use their
category's <code>unk.def</code>
+ template first, with <code>DEFAULT</code> used
when no category template exists.
+ </para>
</section>
</section>
<section xml:id="tools.tokenizer.subword">
