[
https://issues.apache.org/jira/browse/STANBOL-795?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Rupert Westenthaler resolved STANBOL-795.
-----------------------------------------
Resolution: Fixed
implementation provided with http://svn.apache.org/viewvc?rev=1406164&view=rev
> OpenNLP Tokenizer Engine
> ------------------------
>
> Key: STANBOL-795
> URL: https://issues.apache.org/jira/browse/STANBOL-795
> Project: Stanbol
> Issue Type: Sub-task
> Components: Enhancer
> Reporter: Rupert Westenthaler
> Assignee: Rupert Westenthaler
>
> Implement an separate OpenNLP Tokenizer Engine.
> While some Engines like the OpenNLP POS or the CELI Lemmatizer engine do
> support tokenizing (if tokens do not already exist in the Analyzed Text) it
> is important to implement an engine explicitly for this task.
> This engine also supports the language configuration (see following example)
> en;model=SIMPLE
> de;model=mySpecificTokenizerModel_de.bin
> !jp
> !zh
> *
> the 'model' parameter can be used to load specific tokenizer models. "SIMPLE"
> forces the use of the OpenNLP SimpleTokenizer. If no model configuration is
> present the default tokenizer for the language is loaded ("{lang}-token.bin"
> or the simple tokenizer if the language model is not present).
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
