[
https://issues.apache.org/jira/browse/TIKA-2790?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16839413#comment-16839413
]
Tim Allison edited comment on TIKA-2790 at 5/14/19 1:33 PM:
------------------------------------------------------------
[^langid_20190514.zip] includes (dense) confusion matrices, which show that
OpenNLP is detecting "che" incorrectly for quite a few languages at 100k
characters with 0 noise (afr, ast, bos, bre, ceb, cym, dan, deu, ekk, eng, epo,
est, eus, fin, fra...)
My perception based on this info so far:
* OpenNLP is much, much slower than the other options, but it has good
coverage, is robust to noise and the confidence scores track neatly to length
and noise; if we integrate it, the cost in time can be limited by restricting
samples to, say 10k or less?-- we do need to figure out what's going on at
100k...
* Yalder is by far the fastest and has good coverage; it is very sensitive to
noise, and the confidence scores do not track neatly with length and/or noise.
* Optimaize -- fairly fast, covers far fewer languages (although we could add
them), confidence scores track poorly with length and noise.
was (Author: [email protected]):
[^langid_20190514.zip] includes (dense) confusion matrices.
My perception based on this info so far:
* OpenNLP is much, much slower than the other options, but it has good
coverage, is robust to noise and has sane confidence scores -- we do need to
figure out what's going on at 100k...
* Yalder is by far the fastest and has good coverage; it is very sensitive to
noise, and the confidence scores do not track neatly with length and/or noise.
* Optimaize -- fairly fast, covers far fewer languages (although we could add
them), confidence scores track poorly with length and noise.
> Consider switching lang-detection in tika-eval to open-nlp
> ----------------------------------------------------------
>
> Key: TIKA-2790
> URL: https://issues.apache.org/jira/browse/TIKA-2790
> Project: Tika
> Issue Type: Improvement
> Reporter: Tim Allison
> Priority: Major
> Attachments: langid_20190509.zip, langid_20190510.zip,
> langid_20190514.zip
>
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
