[
https://issues.apache.org/jira/browse/TIKA-2790?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16837289#comment-16837289
]
Joern Kottmann commented on TIKA-2790:
--------------------------------------
I worked on the OpenNLP Language Detector, and also used it in production
systems.
The detector is rather simple, and can predict the language for an input
string, there are a few constrains which are worth considering:
- Will fail for very short strings (less than three words), in that case it is
better not to use it at all
- Long strings (e.g. the 100k char string from above) are not increasing the
accuracy much compared to a few sentences, they should be avoided as well, a
shorter string should be passed instead
- Only one language can be detected
A common approach is to implement "probing", where you take random samples from
the input string, detect the language for each string, and then combine the
detected languages. I used this approach for multilingual texts and it
performed well.
> Consider switching lang-detection in tika-eval to open-nlp
> ----------------------------------------------------------
>
> Key: TIKA-2790
> URL: https://issues.apache.org/jira/browse/TIKA-2790
> Project: Tika
> Issue Type: Improvement
> Reporter: Tim Allison
> Priority: Minor
> Attachments: langid_20190509.zip
>
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
