[
https://issues.apache.org/jira/browse/TIKA-354?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14342529#comment-14342529
]
Ken Krugler commented on TIKA-354:
----------------------------------
Better speed is still important, as a 2x improvement from TIKA-1549 is good but
means that now it's only 45% of the web crawl time that's spent determining the
language, versus 90%. However the right way to do this is (with a new detector
library) internally sampling until the target confidence is reached, versus the
caller having to decide how much text to analyze.
So net-net, yes I think this can be closed.
> ProfilingHandler should take a length-limiting parameter
> --------------------------------------------------------
>
> Key: TIKA-354
> URL: https://issues.apache.org/jira/browse/TIKA-354
> Project: Tika
> Issue Type: Improvement
> Components: languageidentifier
> Affects Versions: 0.5
> Reporter: Vivek Magotra
> Assignee: Ken Krugler
> Attachments: TIKA-354-2.patch, TIKA-354.patch
>
>
> ProfilingHandler currently parses the entire document (thereby analyzing
> n-grams for the entire doc).
> ProfilingHandler should take a length-limiting parameter that allows a user
> to specify the amount of data that should get analyzed.
> In fact, by default that limit should be set to something like 8K.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
