Author: rwesten
Date: Sat Nov 24 09:54:49 2012
New Revision: 1413162
URL: http://svn.apache.org/viewvc?rev=1413162&view=rev
Log:
STANBOL-733 - Adaped docu to changes in the code
Modified:
stanbol/site/trunk/content/docs/trunk/components/enhancer/engines/entitylinking.mdtext
Modified:
stanbol/site/trunk/content/docs/trunk/components/enhancer/engines/entitylinking.mdtext
URL:
http://svn.apache.org/viewvc/stanbol/site/trunk/content/docs/trunk/components/enhancer/engines/entitylinking.mdtext?rev=1413162&r1=1413161&r2=1413162&view=diff
==============================================================================
---
stanbol/site/trunk/content/docs/trunk/components/enhancer/engines/entitylinking.mdtext
(original)
+++
stanbol/site/trunk/content/docs/trunk/components/enhancer/engines/entitylinking.mdtext
Sat Nov 24 09:54:49 2012
@@ -283,7 +283,7 @@ The _LabelTokenizer_ interface defines o
As it might very likely be the case that users will want to use multiple
LabelTokenizer for different languages the EntityLinkingEngine comes with an
MainLabelTokenizer implementation. It registers itself as LabelTokenizer with
highest possible OSGI 'service.ranking' and tracks all other registered
_LabelTokenizers_.
-So if custom _LabelTokenizers_ register themselves as OSGI service than the
MainLabelTokenizer can forward requests to them. It will do so in the order of
the '<code>service.ranking</code>'s. in addition _LabelTokenizer_ can use the
'<code>enhancer.engines.keywordextraction.labeltokenizer.languages</code>'
property to formally specify the languages they are supporting. This property
does use the language configuration syntax (e.g. "en,de" would include English
and German; "!it,!fr,*" would specify all languages expect Italian and French).
If no configuration is provided than "*" (all languages) is assumed - what is
fine as default as long as _LabelTokenizer_ correctly return NULL for languages
they do not support.
+So if custom _LabelTokenizers_ register themselves as OSGI service than the
MainLabelTokenizer can forward requests to them. It will do so in the order of
the '<code>service.ranking</code>'s. in addition _LabelTokenizer_ can use the
'<code>enhancer.engines.entitylinking.labeltokenizer.languages</code>' property
to formally specify the languages they are supporting. This property does use
the language configuration syntax (e.g. "en,de" would include English and
German; "!it,!fr,*" would specify all languages expect Italian and French). If
no configuration is provided than "*" (all languages) is assumed - what is fine
as default as long as _LabelTokenizer_ correctly return NULL for languages they
do not support.
The MainLabelTokenizer forwards tokenize requests to all available
LabelTokenizer implementations that support a specific language sorted by their
'<code>service.ranking</code>' until the first one does NOT return NULL. If no
LabelTokenizer was found or all returned NULL it will also return NULL.
@@ -308,11 +308,17 @@ This will inject the MainLabelTokenizer
Configuring the NamedEntityLinkingEngine like this ensures that all registered
_LabelTokenizers_ are considered for tokenizing.s_
+#### Simple LabelTokenizer
+
+This is the default implementation of a LabelTokenizer that does not depend on
any external dependencies. This implementation behaves exactly the same as the
[OpenNLP](http://opennlp.apache.org) SimpleTokenizer. It is active by default
and configured to process all languages. It uses an
'<code>service.ranking</code>' of '-1000' so will be typically overwritten by
custom registers implementations.
+
+The main intension of this implementation is to be a reasonable default
ensuring LabelTokenizer support for all languages.
+
#### OpenNLP LabelTokenizer
-This is the default implementation of an LabelTokenizer based on the
[OpenNLP](http://opennlp.apache.org) tokenizer API. Internally it uses the
OpenNLP service to load tokenizer models for languages. If language specific
model is available it uses the OpenNLP SimpleTokenizer implementation. The
_OpenNlpLabelTokenizer_ registers itself with a '<code>service.ranking</code>'
of '-1000' so it will b
+The EntityLinkingEngie also contains an [OpenNLP](http://opennlp.apache.org)
tokenizer API based implementation. As the dependency to OpenNLP and the
Stanbol Commons OpenNLP module are optionally this implementation will only be
active if the
<code>org.apache.stanbol:org.apache.stanbol.commons.opennlp</code> bundle with
an version starting from <code>0.10.0</code> is active.
-The _LabelTokenizerManager_ interface extends the _
+This _LabelTokenizer_ supports the configuration of custom OpenNLP tokenizer
models for specific languages e.g. "de;model=my-de-tokenizermodel.zip;*" would
use a custom model for German and the default models for all other languages.
-The KeywordLinkingEngine will - by default - always use the LabelTokenizer
with the highest "service.ranking" for a given language to tokenize labels. By
default it comes with an OpenNLP based Tokenizer implementation that registers
itself for all languages with a "service.ranking" of "-1000".
+Internally the OpenNLP service to load tokenizer models for languages. That
means that tokenizer models are loaded via the DataFileProvider infrastructure.
For user that means that custom tokenizer models are loaded from the Stanbol
Datafiles directory ({stanbol-working-dir}/stanbol/datafiles).
