[
https://issues.apache.org/jira/browse/TIKA-1723?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Tim Allison updated TIKA-1723:
------------------------------
Attachment: TIKA-1723v2.patch
This is a very, very rough patch that breaks out a new tika-langdetect module.
The idea is to follow the pattern of tika-translate -- only the smallest
footprint necessary in core. For now, there's quite a bit of duplication of
test resources, etc. Eventually we could move Tika's current langdetect
implementation into this module and leave the scaffolding behind, but probably
not until 2.0.
I wouldn't propose doing this all in one patch (integrating Optimaize and doing
all of the internal restructuring), but I wanted to continue roughing out where
we might be headed. If this basically looks good, I'll open a separate ticket
for that, and then [~kkrugler], we could add this.
If [~chrismattmann] or others want to add configurability for language id
through TikaConfig, I think that'd be great and something along these lines
would be a good start.
> Integrate language-detector into Tika
> -------------------------------------
>
> Key: TIKA-1723
> URL: https://issues.apache.org/jira/browse/TIKA-1723
> Project: Tika
> Issue Type: Improvement
> Components: languageidentifier
> Affects Versions: 1.11
> Reporter: Ken Krugler
> Assignee: Ken Krugler
> Priority: Minor
> Attachments: TIKA-1723.patch, TIKA-1723v2.patch
>
>
> The language-detector project at
> https://github.com/optimaize/language-detector is faster, has more languages
> (70 vs 13) and better accuracy than the built-in language detector.
> This is a stab at integrating it, with some initial findings. There are a
> number of issues this raises, especially if [~chrismattmann] moves forward
> with turning language detection into a pluggable extension point.
> I'll add comments with results below.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
