[
https://issues.apache.org/jira/browse/TIKA-2592?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16381667#comment-16381667
]
Andreas Meier commented on TIKA-2592:
-------------------------------------
Thanks for your response [~kkrugler]
You are right, "unicode" is a supported charset name in Java.
I don't think we should treat unicode as a suspicious charset name in general,
just for html since there is a difference between java and html.
Therefore I think the HtmlEncodingDetector.java is the right place to catch the
"unicode" string and handle it.
That's why I changed HtmlEncodingDetector and not CharsetUtils.java.
Maybe I wasn't clear enough about {quote}All unknown encodings should be
validated or at least be set to default utf-8.{quote}
I thought of "All charset-encodings parsed from html itself should be validated
(and be set to default utf-8 if they are invalid and a proper charset could not
be detected)".
What I know is: we can't rely on charset encoding in meta-tags
> HTML with charset unicode handled as utf-16 instead utf-8
> ---------------------------------------------------------
>
> Key: TIKA-2592
> URL: https://issues.apache.org/jira/browse/TIKA-2592
> Project: Tika
> Issue Type: Bug
> Affects Versions: 2.0, 1.16, 1.17
> Reporter: Andreas Meier
> Priority: Major
> Attachments: TestCharsetUnicodeHTML.html,
> fix-for-TIKA2592-contributed-by-Andreas-Meier.patch
>
>
> HTML files are detected as utf-16 when meta content is set to "unicode".
> {code:XML}
> <meta http-equiv="Content-Type" content="text/html; charset="unicode">
> {code}
>
> Shouldn't the default be utf-8?
> The attached sample file is shown correctly in:
> Chromium Version 55.0.2883.75
> Firefox 50.1.0
> IE 11
> I am aware that there is no charset "unicode" (available character encodings:
> [http://www.iana.org/assignments/character-sets/character-sets.xhtml|http://www.iana.org/assignments/character-sets/character-sets.xhtml])
> Unfortunately there are many wrong encodings used out there.
> All unknown encodings should be validated or at least be set to default utf-8.
> Regards
> Andreas
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
