[
https://issues.apache.org/jira/browse/TIKA-357?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12801679#action_12801679
]
Ken Krugler commented on TIKA-357:
----------------------------------
Hi Chris,
This patch fixes an issue with detecting the charset for an HTML document, not
detecting whether a document is HTML vs. plain text.
It changes the amount of text to "sniff" when trying to find a <meta> tag that
contains charset info for an http-equiv value.
You've run into a separate issue that I've seen, where the AutoDetection code
fails on pages that have big Javascript blocks at the beginning.
> Increase buffer size for meta tag sniffing
> ------------------------------------------
>
> Key: TIKA-357
> URL: https://issues.apache.org/jira/browse/TIKA-357
> Project: Tika
> Issue Type: Improvement
> Affects Versions: 0.5
> Reporter: Ken Krugler
> Assignee: Chris A. Mattmann
> Priority: Minor
> Fix For: 0.6
>
> Attachments: makler.html, TIKA-357.patch
>
>
> Some web pages (such as makler.su, see attached) have lots of script data
> before the body of the HTML.
> When this happens, the sniffing code fails to find the charset info in the
> meta tag, because it currently only sniffs the first 4K.
> Bumping it to 8K would cover all of the cases that I (Ken) have seen during a
> test crawl.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
