[
https://issues.apache.org/jira/browse/TIKA-354?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ken Krugler updated TIKA-354:
-----------------------------
Attachment: TIKA-354-2.patch
Additional improvement for language identification. This patch has to be
applied after TIKA-354.patch.
Use the ratio of best/next best language distances as the key factor, since
this is much less sensitive to the length of the document being analyzed. Added
a test to ensure that a document from a language currently not handled
(Chinese, in this case) fails due to the (now much higher) absolute distance
cut-off.
> ProfilingHandler should take a length-limiting parameter
> --------------------------------------------------------
>
> Key: TIKA-354
> URL: https://issues.apache.org/jira/browse/TIKA-354
> Project: Tika
> Issue Type: Improvement
> Components: languageidentifier
> Affects Versions: 0.5
> Reporter: Vivek Magotra
> Assignee: Ken Krugler
> Attachments: TIKA-354-2.patch, TIKA-354.patch
>
>
> ProfilingHandler currently parses the entire document (thereby analyzing
> n-grams for the entire doc).
> ProfilingHandler should take a length-limiting parameter that allows a user
> to specify the amount of data that should get analyzed.
> In fact, by default that limit should be set to something like 8K.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
