[
https://issues.apache.org/jira/browse/TIKA-1723?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14723347#comment-14723347
]
Tim Allison commented on TIKA-1723:
-----------------------------------
Agreed on complexity of multilingual lang id. You would definitely want to do
a two-step process...I would think from a statistical perspective as well as
from the perspective you point out: majority lang for the doc, etc.
This was in the back of my mind as an "eventually, wouldn't this be nice," but
I think we're still a good way away from that.
Apologies for my lack of clarity, what I meant by "dual language detection and
content handling" was: allow for identification of the overall language of the
document at the same time that you are handling/writing out regular content,
say, to an outputstream or a byte buffer via the usual ToTextHandler (or
friend). I realize that you can probably do this via some kind of wrapping of
the writer, but it seems like we might want to move this into the handler.
> Integrate language-detector into Tika
> -------------------------------------
>
> Key: TIKA-1723
> URL: https://issues.apache.org/jira/browse/TIKA-1723
> Project: Tika
> Issue Type: Improvement
> Components: languageidentifier
> Affects Versions: 1.11
> Reporter: Ken Krugler
> Assignee: Ken Krugler
> Priority: Minor
> Attachments: TIKA-1723.patch, TIKA-1723v2.patch
>
>
> The language-detector project at
> https://github.com/optimaize/language-detector is faster, has more languages
> (70 vs 13) and better accuracy than the built-in language detector.
> This is a stab at integrating it, with some initial findings. There are a
> number of issues this raises, especially if [~chrismattmann] moves forward
> with turning language detection into a pluggable extension point.
> I'll add comments with results below.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
