[
https://issues.apache.org/jira/browse/STANBOL-740?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13500083#comment-13500083
]
Rupert Westenthaler commented on STANBOL-740:
---------------------------------------------
## Linking Process:
The Linking Process consists of three major steps: First it consumes results of
the NLP processing to determine tokens - words - that need to be linked with
the configured vocabulary. Second the linking of entities based on their labels
with the current section of the Text and third the writing of the enhancement
results.
### Token Types
The KeywordLinkingEngine operates based on tokens (words). Those tokens are
divided in the following Categories
* __Linkable Tokens__: This are words that are linked with the Vocabulary. This
means that the engine will issue quires in the controlled vocabulary for those
tokens
* __Matchable Tokens__: Matchable tokens are used to refine quires. For the
matching of entity labels with the text those words are treated in the same way
as linkable words. So the main difference is that matchable words alone will
not cause the engine to query for Entities in the Controlled Vocabulary.
* __Other Tokens__: All other tokens in the text are not used for searches in
the configured vocabulary. However during the matching of labels with the Text
they are considered as they might also be present in labels of entities
"University of Salzburg" is a good example as 'University' - a common noun -
can be considered a matchable token, 'of' an other- and 'Salzburg' as proper
noun is a typical linkable token. As the engine only queries for linkable token
a single query for 'Salzburg' would be issued against the vocabulary. However
this query would also use the matchable token 'University' as a secondary query
term. The token 'of' would only be considered during matching.
In addition to the token type the engine also determines the rolling parameters
* __Token Length__: The number of characters of a word. This is especially
important for languages where no POS tagger is available.
* __Alpha-Numeric__: If a Token does contain an alpha or an numeric character.
This is mainly used to skip processing of tokens that represent punctuation.
* __Upper Case__: Upper Case Tokens do often represent named entities. because
of that the Engine keeps track of upper case words.
* __Token Phrase__: If a Token is member of a _processable_ Phrase. Phrases are
groups of Tokens that can be detected by a Chunker. A typical examples are Noun
Phrases.
### Consumed NLP Processing Results:
The KeywordLinkingEngine consumes NLP processing results from the AnalyzedText
ContentPart of the processed ContentItem. The following list describes the
consumed information and their usage in the linking process:
1. __Language_ _(required)_: The Language of the Text is acquired from the
Metadata of the ContentItem. It is required to search for labels in the correct
language and also to correctly apply language specific configurations of the
engine.
2. __Sentences__ _(optional)_: Sentence annotations are used as segments for
the matching process. In addition for the first word of an Sentence the _Upper
Case_ feature is NOT set. In the case that no Sentence Annotations are present
the whole text is treated as a single Sentence.
3. __Tokens__ _(required)_: As this Engine is based on the processing of Tokens
such information are absolutely required.
4. __POS Annotations__ _(optional)_: Part of Speech (POS) tags are used to
determine the _Token Type_. The NLP processing module provides two enumerations
that define POS types. The high level _Lexical Categories_ (16 members
including "Noun", "Verb", "Adjective", "Adposition" ...) and the Pos
enumeration with ~150 very detailed POS definitions (such as (e.g.
"ProperNoun", "CommonNoun", "Infinitive", "Gerund", "PresentParticiple" …). In
addition the engine can also be configured to use the string tag as used by the
POS tagger. The mapping of the _POS Annotation_ to the _Token Type_ is provided
by the Engine configuration and can be language specific.
5. __Phrase Annotation__ _(optional)_: Phrase Annotations of Chunks present in
the AnalyzedText are checked against the configured processable phrase
categories. The linking of Tokens is NOT limited to Tokens within processable
phrases. Phrases are only used as additional context to improve the matching
process. The _Lexical Category_ and the string tags used by the Chunker can be
used to configure the processable Phrase categories.
6. __Lemma__ _(optional)_: The Lemma provided by the MorphoAnalysis annotation
can be used for linking instead of the token as used within the text.
### Entity Linking:
The linking process is based the matching of labels of entities returned as
result for searches for entities in the configured controlled vocabulary. In
addition the engine can be configured to consider redirects for entities
returned by searches.
Searches are issued only for _Linkable Tokens_ and may include up to _Max
Search Tokens_ additional _Linkable-_ or _Matchable Tokens_. If the _Linkable
Token_ is within an _Phrase_ than only other tokens within the same phrase are
considered. Otherwise any _Linkable-_ or _Matchable Tokens_ within the
configured _Max Search Token Distance_ is considered for the search.
Searches to the controlled vocabulary are issued using the _EntitySearcher_
interface and build like follows:
{lt}@{lang} || {lt}@{dl} || [{at}@{lang} || {at}@{dl} ... ]
where:
* {lt} ... the _Linkable Token_ for that the search is issued
* {at} ... additional _Linkable-_ or _Matchable Tokens_ included in the
search
* {lang} ... the language of the text
* {dl} ... the configured _Default Matching Language_. If {df} == {lang}
than the or term(s) for the {dl} are omitted
For results of those queries the labels in the {lang} and {dl} are matched
against the text. However {dl} labels are only considered if no match was found
for labels in the language of the text. For matching labels with the Tokens of
the text the engine need to tokenize the labels. This is done by using the
_LabelTokenizer_ interface.
The matching process distinguishes between matchable and non-matchable Tokens
as well as non-alpha-numeric Tokens that are completely ignored. Matching
starts at the position of the _Linkable Token_ for that the search in the
configured vocabulary was issued. From this position Tokens in the Label are
matched with Tokens in the text until the first matchable or 2nd non-matchable
token is not found. In a second round the same is done in the backward
direction. The configured _Min Token Match Factor_ determines how exact tokens
in the text must correspond to tokens in the label so that a match is
considered. This is repeated for all labels of an Entity. The label match that
covers the most tokens is than considered as the match for that Entity.
There are various parameters that can be used to fine tune the matching
process. But the most important decision is if one want to include suggestions
where labels with two tokens do only match a single _Matchable Token_ in the
Text (e.g. "Barack Obama" matching "Obama" but also 1000+ "Tom {something}"
matching "Tom"). The default configuration of the Engine excludes those but
depending on the use case and the linked vocabulary users might want to change
this. See the documentation of the _Min Matched Tokens_ and _Min Label Match
Score_ for details and examples.
### Writing Enhancement Results
This step covers the following steps:
* processing of redirects as configured by the _Redirect Mode_
* mapping of the Entity types to the dc:type values for fise:TextAnnotations as
configured by the _Type Mappings_ configuration
* if _Dereference Entities_ is enabled than information for all configured
_Dereferenced Fields_ need to be obtained
* writing of the fise:TextAnnotations, fise:EntityAnnotations and dereferenced
entities (if enabled) to the metadata of the processed ContentItem
> Adopt the KeywordLinkingEngine to use the AnalyzedText content part
> -------------------------------------------------------------------
>
> Key: STANBOL-740
> URL: https://issues.apache.org/jira/browse/STANBOL-740
> Project: Stanbol
> Issue Type: Sub-task
> Reporter: Rupert Westenthaler
> Assignee: Rupert Westenthaler
>
> The KeywordLinkingEngine currently does both NLP processing AND linking
> against the target vocabulary. Up to now this was the only possibility as
> separating those two things was not feasible with the limitations of the RDF
> metadata.
> With the introduction of the AnalyzedText content part the NLP processing
> part needs no longer be part of the KeywordLinkingEngine.
> This issue covers
> * removal of the NLP related functionality from the KeywordLinkingEngine
> * reimplementation of the linking part on top of the API provided by the
> AnalyzedText contentpart
> * add support fore new features of the NLP chain
> * use lemmas - if available - for entity lookup
> * use POS tagset mappings to the OLIA ontology to decide what tokens to
> lookup
> After this change the KeywordLinkingEngine will also be able to work in
> combination with any NLP framework that is integrated with the Stanbol NLP
> components (writes its data to the AnalyzedText content part).
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
