This is an automated email from the ASF dual-hosted git repository.
krickert pushed a change to branch OPENNLP-1894-lattice-cjk
in repository https://gitbox.apache.org/repos/asf/opennlp.git
from db5f6cc06 OPENNLP-1894: Trim parsed lines as Unicode whitespace and
document the tokenizer overrides
add b02948cfc OPENNLP-1894: Address review: complete javadoc, hoist
constants, and fold fixture duplication
No new revisions were added by this update.
Summary of changes:
dev/README-mecab-dictionaries.md | 2 +-
.../tools/tokenize/lattice/LatticeTokenizer.java | 65 +++--
.../tools/tokenize/lattice/MecabDictionary.java | 140 ++++++++---
.../tokenize/lattice/MecabDictionaryInstaller.java | 10 +-
.../tools/tokenize/lattice/UnigramSegmenter.java | 32 ++-
.../tokenize/lattice/LatticeTokenizerTest.java | 276 ++++++++++++++-------
.../lattice/MecabDictionaryInstallerTest.java | 2 +
.../tools/tokenize/lattice/TarGzArchives.java | 48 ++--
.../tokenize/lattice/UnigramSegmenterTest.java | 32 ++-
9 files changed, 424 insertions(+), 183 deletions(-)
