On 11/9/11 10:57 AM, Jens Grivolla wrote:
Hi,
inspired by the discussion started on the OpenNLP list (see
http://mail-archives.apache.org/mod_mbox/incubator-opennlp-dev/201111.mbox/%3CCAE%3D29DrYQ1YeVdQVF_Qp-6aKnLubGk1r0F-Sk5Ttd8viK0c5SQ%40mail.gmail.com%3E
) I would like to take the opportunity to get some feedback here.
OpenNLP will not really work if you filter out certain tokens, I can
understand that this behavior makes sense
for Lucene but for most text analysis which is usually done by UIMA
components (pos tagging, ner, parsing, etc.) it does not.
We are starting (slowly, and with very limited resources) to work on
integrating the tokenizers from Lucene/Solr into UIMA (see
http://wiki.apache.org/solr/AnalyzersTokenizersTokenFilters). This
would give us a quite powerful framework for tokenization including
filtering layers, etc., as well as a jFLEX based engine for defining
tokenizers. We're hoping to be able to reproduce the XML configuration
used in Solr that lets you define character filters, tokenizers, token
filters, etc.
Do you want to take an input text and produce a second "filterd" sofa,
or would these be added as annotations to the CAS?
The UIMA concept is different to the one we have in Lucene. The idea in
UIMA is to enhance a sofa with more and more analysis
data step by step. And in Lucene they transform the input data until it
fits their need.
One issue that also came up on the thread on the OpenNLP list is that
Lucene/Solr tokenizers normally skip punctuation tokens, which are
irrelevant for search but important for many other tasks. From what we
have seen this could easily be fixed in the corresponding jFlex grammar.
We would also need to integrate sentence splitting. We are thinking of
implementing a filter that detects abbreviations, emoticons, etc. and
use remaining punctuation tokens as sentence boundaries.
The OpenNLP sentence splitter works great and is already integrated into
UIMA. Would using it be an option for you?
I am using it to process news articles in various languages together
with UIMA-AS.
Jörn
