Andreas Meier commented on TIKA-2592:

Before making this kind of change (default "unicode" to UTF-8), wouldn't we 
want to determine how many HTML documents use "unicode" as the charset and are 
UTF-16 vs. UTF-8?

Good point, my suggestion to set UTF-8 as default was a bit 
short-sighted.(That's why I like discussions)

I'm not sure we should make a rule "unicode->UTF-8", rather we should ignore 
"unicode" if it is in a metaheader and let the charsetdetector do its work? 
What do you think?

This might probably be the best solution for the problem,
Ignoring meta charset="unicode" solves the problem at least for the testfile.
What do you think about Tims suggestion [~kkrugler]?

Do you have a testcorpus or are you crawling the web [~talli...@mitre.org]?

> HTML with charset unicode handled as utf-16 instead utf-8
> ---------------------------------------------------------
>                 Key: TIKA-2592
>                 URL: https://issues.apache.org/jira/browse/TIKA-2592
>             Project: Tika
>          Issue Type: Bug
>    Affects Versions: 2.0, 1.16, 1.17
>            Reporter: Andreas Meier
>            Priority: Major
>         Attachments: TestCharsetUnicodeHTML.html, 
> fix-for-TIKA2592-contributed-by-Andreas-Meier.patch
> HTML files are detected as utf-16 when meta content is set to "unicode".
> {code:XML}
> <meta http-equiv="Content-Type" content="text/html; charset="unicode">
>  {code}
> Shouldn't the default be utf-8?
> The attached sample file is shown correctly in:
> Chromium Version 55.0.2883.75
> Firefox 50.1.0
> IE 11
> I am aware that there is no charset "unicode" (available character encodings: 
> [http://www.iana.org/assignments/character-sets/character-sets.xhtml|http://www.iana.org/assignments/character-sets/character-sets.xhtml])
> Unfortunately there are many wrong encodings used out there.
> All unknown encodings should be validated or at least be set to default utf-8.
> Regards 
> Andreas

This message was sent by Atlassian JIRA

Reply via email to