[
https://issues.apache.org/jira/browse/TIKA-4690?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Tim Allison resolved TIKA-4690.
-------------------------------
Fix Version/s: 4.0.0
Resolution: Fixed
We eventually took this out I think. I'm closing this ticket because the work
was completed.
> Add generative language model in 4.x
> ------------------------------------
>
> Key: TIKA-4690
> URL: https://issues.apache.org/jira/browse/TIKA-4690
> Project: Tika
> Issue Type: Task
> Reporter: Tim Allison
> Priority: Minor
> Fix For: 4.0.0
>
>
> Finally realized that we can play all we want with logits from the language
> detector, but it is not a great approach for "languagey/junk" detection. On
> this ticket, we'll add a generative model trained on the same languages as
> the language detector so that we can get a better sense of, for example,
> "Lang detector said Thai, how likely is it to actually be Thai?"
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
