This is an automated email from the ASF dual-hosted git repository. krickert pushed a commit to branch OPENNLP-1894-lattice-cjk in repository https://gitbox.apache.org/repos/asf/opennlp.git
commit 104cd0e2954c30a9bed1a157ba7ae38341fef572 Author: Kristian Rickert <[email protected]> AuthorDate: Mon Jul 20 04:44:21 2026 -0400 OPENNLP-1894: Document lattice CJK tokenization with a mirror-tested example Add a lattice tokenizer section to the manual citing LatticeUsageExampleTest. --- opennlp-docs/src/docbkx/tokenizer.xml | 25 +++++++++++++++++++++++++ 1 file changed, 25 insertions(+) diff --git a/opennlp-docs/src/docbkx/tokenizer.xml b/opennlp-docs/src/docbkx/tokenizer.xml index cd1d8a2dd..fbc74fbb2 100644 --- a/opennlp-docs/src/docbkx/tokenizer.xml +++ b/opennlp-docs/src/docbkx/tokenizer.xml @@ -538,5 +538,30 @@ wordTokenizer.tokenize("The quick brown fox.", (start, end, type) -> { </programlisting> </para> </section> + <section xml:id="tools.tokenizer.lattice"> + <title>Lattice tokenization for CJK</title> + <para> + Languages written without spaces need a dictionary-backed segmenter. + <code>LatticeTokenizer</code> scores paths over a mecab-format dictionary and + emits the cheapest segmentation with spans in original text coordinates. + <code>UnigramSegmenter</code> does the same from a plain frequency lexicon. + Install a dictionary archive with <code>MecabDictionaryInstaller</code>, load it + as a <code>MecabDictionary</code>, and tokenize. + <code>LatticeUsageExampleTest</code> asserts the install-load-tokenize and + lexicon flows shown here. + <programlisting language="java"> +<![CDATA[MecabDictionaryInstaller.install(archiveUri, dictionaryDirectory); +LatticeTokenizer tokenizer = + new LatticeTokenizer(MecabDictionary.load(dictionaryDirectory)); + +String[] tokens = tokenizer.tokenize(text); +Span[] spans = tokenizer.tokenizePos(text); +List<Morpheme> morphemes = tokenizer.analyze(text); + +UnigramSegmenter segmenter = UnigramSegmenter.load(Path.of("words.txt")); +String[] words = segmenter.tokenize(text);]]> + </programlisting> + </para> + </section> </section> </chapter>
