[
https://issues.apache.org/jira/browse/TIKA-2790?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Tim Allison resolved TIKA-2790.
-------------------------------
Resolution: Fixed
Assignee: Tim Allison
Fix Version/s: 1.22
More work remains for short and/or noisy text, but the work performed on this
ticket:
* Increases the number of langs detected from 75 to 121
* Increases the common words lists from 21 languages to 121 languages
* Improves processing speed over Optimaize on un-noised data > 1000 characters
Evaluation code is available:
https://github.com/tballison/tika-addons/tree/master/lang-id
> Consider switching lang-detection in tika-eval to open-nlp
> ----------------------------------------------------------
>
> Key: TIKA-2790
> URL: https://issues.apache.org/jira/browse/TIKA-2790
> Project: Tika
> Issue Type: Improvement
> Reporter: Tim Allison
> Assignee: Tim Allison
> Priority: Major
> Fix For: 1.22
>
> Attachments: fra_mixed_100000_0.0_0.txt, hasEnough.png,
> langid_20190509.zip, langid_20190510.zip, langid_20190514.zip,
> langid_20190514_plus_minus_1.zip, rollups_20190716.zip, timeVsLength.png
>
>
--
This message was sent by Atlassian JIRA
(v7.6.14#76016)
