krickert opened a new pull request, #1191: URL: https://github.com/apache/opennlp/pull/1191
Adds Viterbi lattice segmentation over user-supplied MeCab-format dictionaries, covering the Japanese (IPADIC) and Korean (mecab-ko-dic) dictionary layouts, plus a frequency-driven unigram segmenter for Chinese. The lexicon is held in a double-array trie with frequency-recoded labels, and lattice nodes chain intrusively so the hot path allocates no per-position lists. Measured on the real IPADIC dictionary: about 5 million characters per second single-threaded, with 392k entries loading in under one second. Segmentation of the standard connection-cost-sensitive test sentences matches the reference implementation's output on IPADIC. Dictionaries are always user-supplied and never bundled; test fixtures are project-authored miniature lexicons written by the tests themselves, and `dev/README-mecab-dictionaries.md` plus a checksum-verifying download script document dictionary acquisition. The tokenizer manual gains a section whose example is asserted by `LatticeUsageExampleTest`. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
