Michael Baessler wrote:
Hi,
I'm one of the Apache UIMA committers and while searching for an open
source language detection technology I found the
Nutch LanguageIdentifierPlugin.
Hello Michael,
Now my questions:
Is there a place where I can find some more details about how your
language identification works?
It uses character n-gram models of different languages, i.e. histograms
of relative frequencies of character groups. It builds a similar model
for the text under examination, and then compares its model to other
pre-defined models. The best match wins. This method is described in a
paper by Cavnar and Trenkle (http://citeseer.ist.psu.edu/68861.html).
This works very well even for short texts, and doesn't require any
linguistic knowledge. However, it works poorly for texts that contain
sections in different languages, or texts in an unknown language, or
extremely short texts.
Will it be possible to share the language identification technology so
that we can wrap it as UIMA analysis engine? My current understanding
is, that it is only available within Nutch but not separately.
There is a grass-roots effort underway to extract portions of Nutch
related to content parsing into a separate framework, called Tika. Jukka
Zitting and Chris Mattmann would be the right people to talk to.
Since both projects are hosted on Apache, I don't see any license issues
when using your technology. :-)
Neither do I. AFAIK, ASF encourages maximum re-use of Apache components
over external ones.
--
Best regards,
Andrzej Bialecki <><
___. ___ ___ ___ _ _ __________________________________
[__ || __|__/|__||\/| Information Retrieval, Semantic Web
___|||__|| \| || | Embedded Unix, System Integration
http://www.sigram.com Contact: info at sigram dot com
