[
https://issues.apache.org/jira/browse/TIKA-2790?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16856790#comment-16856790
]
Tim Allison commented on TIKA-2790:
-----------------------------------
The overlap in the top 30k tokens >= 3 characters per language not surprisingly
follows similar patterns to the above confusion matrix.
||langA||langB||percent||
|bos|hrv|75%|
|fas|pes|74%|
|ind|sun|73%|
|ceb|war|55%|
|ekk|est|53%|
|ind|msa|53%|
|glg|spa|52%|
|lav|lvs|52%|
|nno|nob|51%|
|bak|tat|48%|
> Consider switching lang-detection in tika-eval to open-nlp
> ----------------------------------------------------------
>
> Key: TIKA-2790
> URL: https://issues.apache.org/jira/browse/TIKA-2790
> Project: Tika
> Issue Type: Improvement
> Reporter: Tim Allison
> Priority: Major
> Attachments: fra_mixed_100000_0.0_0.txt, hasEnough.png,
> langid_20190509.zip, langid_20190510.zip, langid_20190514.zip,
> langid_20190514_plus_minus_1.zip, timeVsLength.png
>
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
