This is an automated email from the ASF dual-hosted git repository. krickert pushed a commit to branch OPENNLP-1894-lattice-cjk in repository https://gitbox.apache.org/repos/asf/opennlp.git
commit 759af0e64233d6e560939f333c647dd56778e003 Author: Kristian Rickert <[email protected]> AuthorDate: Thu Jul 16 01:32:44 2026 -0400 OPENNLP-1894: Document dictionary acquisition with a checksum-verifying download helper --- .../dev/README-mecab-dictionaries.md | 83 ++++++++++++++++++++++ .../dev/download-mecab-dictionary.sh | 69 ++++++++++++++++++ 2 files changed, 152 insertions(+) diff --git a/opennlp-core/opennlp-runtime/dev/README-mecab-dictionaries.md b/opennlp-core/opennlp-runtime/dev/README-mecab-dictionaries.md new file mode 100644 index 000000000..c3244ba9c --- /dev/null +++ b/opennlp-core/opennlp-runtime/dev/README-mecab-dictionaries.md @@ -0,0 +1,83 @@ +<!-- + Licensed to the Apache Software Foundation (ASF) under one or more + contributor license agreements. See the NOTICE file distributed with + this work for additional information regarding copyright ownership. + The ASF licenses this file to You under the Apache License, Version 2.0 + (the "License"); you may not use this file except in compliance with + the License. You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + + Unless required by applicable law or agreed to in writing, software + distributed under the License is distributed on an "AS IS" BASIS, + WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + See the License for the specific language governing permissions and + limitations under the License. +--> + +# CJK dictionaries for the lattice tokenizer + +The lattice tokenizer (`opennlp.tools.tokenize.lattice`) segments Japanese and Korean over a mecab-format dictionary, and the unigram segmenter handles Chinese over a plain word-frequency lexicon. Apache OpenNLP bundles no dictionary data: you download a dictionary from the project of your choice, and by using it you accept that dictionary's license. Read the license file inside the archive before use; nothing in this repository grants you those terms. + +## Known mecab-format dictionary projects + +| Dictionary | Language | Archive to download | Encoding of the files | +|---|---|---|---| +| IPADIC 2.7.0 | Japanese | `mecab-ipadic-2.7.0-20070801.tar.gz` from the MeCab project's SourceForge file area (`sourceforge.net/projects/mecab/files/mecab-ipadic/2.7.0-20070801/`) | EUC-JP | +| mecab-ko-dic 2.1.1 | Korean | `mecab-ko-dic-2.1.1-20180720.tar.gz` from the project's Bitbucket downloads page (`bitbucket.org/eunjeon/mecab-ko-dic/downloads/`) | UTF-8 | + +Both archives are gzip-compressed tars, the format `MecabDictionaryInstaller` reads. The encoding column matters: `MecabDictionary.load(Path)` assumes UTF-8, and a dictionary in any other encoding is loaded with the two-argument overload, for example `MecabDictionary.load(dir, Charset.forName("EUC-JP"))` for IPADIC. + +## Step 1: download the archive + +Either fetch the archive with any tool you like, or use the helper next to this file, which adds checksum verification: + +``` +./download-mecab-dictionary.sh <archive-url> ipadic.tar.gz [expected-sha256] +``` + +On the first run without a checksum the script prints the SHA-256 it computed; record it and pass it on later runs so a changed or corrupted download fails loudly instead of being installed. + +## Step 2: unpack with the installer + +The installer extracts only the files a `MecabDictionary` reads (`*.csv`, `*.def`, and `dicrc`), flattens them into the target directory, and by the same flattening makes it impossible for an archive path to escape that directory: + +```java +import java.nio.file.Path; +import opennlp.tools.tokenize.lattice.MecabDictionaryInstaller; + +int files = MecabDictionaryInstaller.install( + Path.of("ipadic.tar.gz").toUri(), Path.of("ipadic")); +``` + +The returned count is the number of dictionary files extracted. A remote URI works in the same call if you prefer to skip step 1 entirely and let the installer stream the download. + +## Step 3: load and tokenize + +```java +import java.nio.charset.Charset; +import java.nio.file.Path; +import opennlp.tools.tokenize.lattice.LatticeTokenizer; +import opennlp.tools.tokenize.lattice.MecabDictionary; + +MecabDictionary dictionary = + MecabDictionary.load(Path.of("ipadic"), Charset.forName("EUC-JP")); +LatticeTokenizer tokenizer = new LatticeTokenizer(dictionary); +String[] tokens = tokenizer.tokenize("東京都に行く"); +``` + +For a UTF-8 dictionary such as mecab-ko-dic, `MecabDictionary.load(Path.of("ko-dic"))` is enough. Loaded dictionaries and tokenizers are immutable and safe to share between threads, so load once and reuse. + +## Chinese: the unigram segmenter needs only a frequency lexicon + +`opennlp.tools.tokenize.lattice.UnigramSegmenter` does not use mecab dictionaries. It loads a plain text lexicon, one entry per line: the word, its count, and optionally a tag, separated by whitespace. Any word-frequency list you have the rights to use works: + +```java +import java.nio.file.Path; +import opennlp.tools.tokenize.lattice.UnigramSegmenter; + +UnigramSegmenter segmenter = UnigramSegmenter.load(Path.of("words.txt")); +String[] tokens = segmenter.tokenize("我来到北京天安门"); +``` + +As with the dictionaries, the lexicon's license is between you and its publisher; nothing is bundled. diff --git a/opennlp-core/opennlp-runtime/dev/download-mecab-dictionary.sh b/opennlp-core/opennlp-runtime/dev/download-mecab-dictionary.sh new file mode 100755 index 000000000..30c5b23bd --- /dev/null +++ b/opennlp-core/opennlp-runtime/dev/download-mecab-dictionary.sh @@ -0,0 +1,69 @@ +#!/usr/bin/env bash +# Licensed to the Apache Software Foundation (ASF) under one or more +# contributor license agreements. See the NOTICE file distributed with +# this work for additional information regarding copyright ownership. +# The ASF licenses this file to You under the Apache License, Version 2.0 +# (the "License"); you may not use this file except in compliance with +# the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +# Downloads a mecab-format dictionary archive for the lattice tokenizer. +# +# This script only fetches the archive and, when a checksum is given, verifies it. +# It does not unpack anything: unpacking is the job of +# opennlp.tools.tokenize.lattice.MecabDictionaryInstaller, which extracts exactly the +# files a MecabDictionary reads and nothing else. See README-mecab-dictionaries.md in +# this directory for the known dictionary projects, their encodings, and the Java +# steps that follow the download. +# +# The dictionary you download carries its own license, which you accept by using it. +# Nothing is bundled with Apache OpenNLP and no dictionary location is built in. + +set -euo pipefail + +usage() { + echo "usage: $0 <archive-url> <target-file> [expected-sha256]" >&2 + echo "" >&2 + echo " archive-url where to fetch the gzip-compressed tar from" >&2 + echo " target-file where to store the downloaded archive" >&2 + echo " expected-sha256 optional checksum; when given, the download is verified" >&2 + echo " and a mismatch deletes the file and fails the script" >&2 + exit 2 +} + +[ $# -lt 2 ] && usage +url="$1" +target="$2" +expected="${3:-}" + +# Download to a temporary name first so an interrupted transfer never leaves a +# half-written file at the target path. +tmp="${target}.download" +echo "downloading ${url}" +curl --fail --location --retry 3 --output "${tmp}" "${url}" + +actual="$(sha256sum "${tmp}" | cut -d' ' -f1)" +if [ -n "${expected}" ]; then + if [ "${actual}" != "${expected}" ]; then + rm -f "${tmp}" + echo "checksum mismatch: expected ${expected}" >&2 + echo " but got ${actual}" >&2 + exit 1 + fi + echo "checksum verified: ${actual}" +else + echo "sha256 of the download: ${actual}" + echo "record this value and pass it as the third argument next time to verify" +fi + +mv "${tmp}" "${target}" +echo "stored ${target}" +echo "" +echo "next: unpack and load it from Java; see README-mecab-dictionaries.md"
