Rupert Westenthaler created STANBOL-795:
-------------------------------------------
Summary: OpenNLP Tokenizer Engine
Key: STANBOL-795
URL: https://issues.apache.org/jira/browse/STANBOL-795
Project: Stanbol
Issue Type: Sub-task
Reporter: Rupert Westenthaler
Assignee: Rupert Westenthaler
Implement an separate OpenNLP Tokenizer Engine.
While some Engines like the OpenNLP POS or the CELI Lemmatizer engine do
support tokenizing (if tokens do not already exist in the Analyzed Text) it is
important to implement an engine explicitly for this task.
This engine also supports the language configuration (see following example)
en;model=SIMPLE
de;model=mySpecificTokenizerModel_de.bin
!jp
!zh
*
the 'model' parameter can be used to load specific tokenizer models. "SIMPLE"
forces the use of the OpenNLP SimpleTokenizer. If no model configuration is
present the default tokenizer for the language is loaded ("{lang}-token.bin" or
the simple tokenizer if the language model is not present).
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
