[
https://issues.apache.org/jira/browse/TIKA-2790?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16839413#comment-16839413
]
Tim Allison commented on TIKA-2790:
-----------------------------------
[^langid_20190514.zip] includes (dense) confusion matrices.
My perception based on this info so far:
* OpenNLP is much, much slower than the other options, but it has good
coverage, is robust to noise and has sane confidence scores -- we do need to
figure out what's going on at 100k...
* Yalder is by far the fastest and has good coverage; it is very sensitive to
noise, and the confidence scores do not track neatly with length and/or noise.
* Optimaize -- fairly fast, covers far fewer languages (although we could add
them), confidence scores track poorly with length and noise.
> Consider switching lang-detection in tika-eval to open-nlp
> ----------------------------------------------------------
>
> Key: TIKA-2790
> URL: https://issues.apache.org/jira/browse/TIKA-2790
> Project: Tika
> Issue Type: Improvement
> Reporter: Tim Allison
> Priority: Major
> Attachments: langid_20190509.zip, langid_20190510.zip,
> langid_20190514.zip
>
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
