[
https://issues.apache.org/jira/browse/TIKA-1443?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14984111#comment-14984111
]
Ken Krugler commented on TIKA-1443:
-----------------------------------
Hi [[email protected]] - I did look at it, and realized I wanted to do
something different in Yalder. Current idea is to use the ratio of known (in a
model) vs. unknown ngrams extracted from the text, plus the relative difference
between the scores of a result and the next best result to decide the
confidence for a result. Tricky part is that an unknown 1-gram wants to have a
lot more impact on the confidence, vs. an unknown 4-gram...setting those
weights properly is a challenge. And I was only using text from the UN
declaration on human rights (in unsupported languages) for trying to set these,
versus true garbage text. Though I'd expect that if I can set valid confidence
for unsupported languages, it should work for garbage.
> Add a junk text detector to Tika
> --------------------------------
>
> Key: TIKA-1443
> URL: https://issues.apache.org/jira/browse/TIKA-1443
> Project: Tika
> Issue Type: Wish
> Reporter: Tim Allison
> Priority: Minor
>
> It would be helpful to have a detector that flags documents whose extracted
> text is junk. This could be used as a component of TIKA-1332 or as a
> standalone detector. See TIKA-1332 for some initial ideas of what statistics
> we might use for such a detector.
> Two use cases:
> * Parser developers could quickly see whether changes in code lead to less
> "junky" documents or more "junky" documents. This would also aid in
> prioritizing manual review of output comparison (see discussion in TIKA-1419).
> * Search system integrators could use that information to set document
> specific relevancy rankings or to avoid indexing a document
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
