[ https://issues.apache.org/jira/browse/TIKA-2592?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16386017#comment-16386017 ]
Andreas Meier edited comment on TIKA-2592 at 3/5/18 12:44 PM: -------------------------------------------------------------- Thanks Tim, but I think I will just download the govdocs1 and work on my own system. (If I find something special there, I will reference it) Too many people and nightly builds stressing one vm may be too much. I already have a small testset I run tika against (~300k+ files), that is also the reason for the numerous tickets I created lately. {quote}That you were referring to the difference between what Java's CharSet class says is a valid name, and the set of names in the IANA standard, right?{quote} Yes {quote}If so, then I agree that what's needed is a list of valid IANA charset names, and we should ignore any charset name in the HTML that's not in this set. Unfortunately Java appears to have a bug, where even though "unicode" isn't a valid IANA charset name, that CharSet returns true from the isRegistered() method. So to make this work, we'd need to roll our own, e.g. using the attached list of names.{quote} Thanks for the charset names upload. I evaluated sun.nio.cs.StandardCharsets which is used to retrieve the java charsets based on a given string. So I tried to determine the strings that do not occur in the IANA list and therefore might reference a wrong java charset. The results are in https://issues.apache.org/jira/secure/attachment/12913030/StandardCharsets_unsupported_by_IANA.txt (I used columns "Preferred MIME Name", "Name" and "Aliases") Posted for completion. (sorry for the bad english...) was (Author: andreasmeier): Thanks Tim, but I think I will just download the govdocs1 and work on my own system. (If I find something special there, I will reference it) Too many people and nightly builds stressing one vm may be too much. I already have a small testset I run tika against (~300k+ files), that is also the reason for the numerous tickets I created lately. {quote}That you were referring to the difference between what Java's CharSet class says is a valid name, and the set of names in the IANA standard, right?{quote} Yes {quote}If so, then I agree that what's needed is a list of valid IANA charset names, and we should ignore any charset name in the HTML that's not in this set. Unfortunately Java appears to have a bug, where even though "unicode" isn't a valid IANA charset name, that CharSet returns true from the isRegistered() method. So to make this work, we'd need to roll our own, e.g. using the attached list of names.{quote} Thanks for the charset names upload. I evaluated sun.nio.cs.StandardCharsets which is used to retrieve the java charsets based on a given string. So I tried to determine the strings that do not occur in the IANA list and therefore might reference a wrong java charset. The results are in https://issues.apache.org/jira/secure/attachment/12913030/StandardCharsets_unsupported_by_IANA.txt (I used columns "Preferred MIME Name", "Name" and "Aliases") On base of that list we should be able to create a set of strings which shall not be resolved by java.nio.charset.Charset forName() Please check the list again to ensure that I did not miss anything. (check sun.nio.cs.StandardCharsets for the actual strings) (sorry for the bad english...) > HTML with charset unicode handled as utf-16 instead utf-8 > --------------------------------------------------------- > > Key: TIKA-2592 > URL: https://issues.apache.org/jira/browse/TIKA-2592 > Project: Tika > Issue Type: Improvement > Affects Versions: 2.0, 1.16, 1.17 > Reporter: Andreas Meier > Priority: Minor > Attachments: IANA Charset names.txt, > StandardCharsets_unsupported_by_IANA.txt, TestCharsetUnicodeHTML.html, > TestHTMLCharsetArabicCP1256.html, TestHTMLCharsetCP1256.html, > fix-for-TIKA2592-contributed-by-Andreas-Meier.patch > > > HTML files are detected as utf-16 when meta content is set to "unicode". > {code:XML} > <meta http-equiv="Content-Type" content="text/html; charset="unicode"> > {code} > > Shouldn't the default be utf-8? > The attached sample file is shown correctly in: > Chromium Version 55.0.2883.75 > Firefox 50.1.0 > IE 11 > I am aware that there is no charset "unicode" (available character encodings: > [http://www.iana.org/assignments/character-sets/character-sets.xhtml|http://www.iana.org/assignments/character-sets/character-sets.xhtml]) > Unfortunately there are many wrong encodings used out there. > All unknown encodings should be validated or at least be set to default utf-8. > Regards > Andreas -- This message was sent by Atlassian JIRA (v7.6.3#76005)