[
https://issues.apache.org/jira/browse/TIKA-2671?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16525028#comment-16525028
]
Gerard Bouchar commented on TIKA-2671:
--------------------------------------
I think the most difficult part of the specification is charset switching, when
a meta that was not detected during prescan is parsed, and the document has to
be reparsed with the new charset. I think this would require changes in the
architecture of tika, because charset detection is currently independent of
parsing.
But without this, I think we can make a decent (although not perfect) charset
detector in tika. See my attempt
[here|https://issues.apache.org/jira/browse/TIKA-2673?focusedCommentId=16520281&page=com.atlassian.jira.plugin.system.issuetabpanels%3Acomment-tabpanel#comment-16520281].
As for testing, the W3C did a much more serious job than I did for testing
browsers. You can have a look at [their results on their
blog|https://www.w3.org/International/tests/repository/html5/the-input-byte-stream/results-basics].
I confirm HTTP headers have priority over meta tags.
> HtmlEncodingDetector doesnt take provided metadata into account
> ---------------------------------------------------------------
>
> Key: TIKA-2671
> URL: https://issues.apache.org/jira/browse/TIKA-2671
> Project: Tika
> Issue Type: Bug
> Components: detector
> Reporter: Gerard Bouchar
> Priority: Major
>
> org.apache.tika.parser.html.HtmlEncodingDetector ignores the document's
> metadata. So when using it to detect the charset of an HTML document that
> came with a conflicting charset specified at the transport layer level, the
> encoding specified inside the file is used instead.
> This behavior does not conform to what is [specified by the W3C for
> determining the character encoding of HTML
> pages|https://html.spec.whatwg.org/multipage/parsing.html#determining-the-character-encoding].
> This causes bugs similar to NUTCH-2599.
> If HtmlEncodingDetector is not meant to take into account meta-information
> about the document, then maybe another detector should be provided, that
> would be a CompositeDetector including, in that order:
> * a new, simple, MetadataEncodingDetector, that would simply return the
> encoding
> * the existing HtmlEncodingDetector
> * a generic detector, like UniversalEncodingDetector
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
