[jira] [Comment Edited] (TIKA-2673) HtmlEncodingDetector doesn't follow the specification

[Gerard Bouchar (JIRA)]

    [ 
https://issues.apache.org/jira/browse/TIKA-2673?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16520137#comment-16520137
 ] 

Gerard Bouchar edited comment on TIKA-2673 at 6/22/18 8:59 AM:
---------------------------------------------------------------

I think at least the following tests should be included, and pass:

{code}
    @Test
    public void bom() throws IOException {
        // A BOM should have precedence over the meta
        // let assertCharset encode the string in the expected charset
        assertCharset("\ufeff<meta charset='WINDOWS-1252'>", 
StandardCharsets.UTF_8);
        assertCharset("\ufeff<meta charset='WINDOWS-1252'>", 
StandardCharsets.UTF_16LE);
        assertCharset("\ufeff<meta charset='WINDOWS-1252'>", 
StandardCharsets.UTF_16BE);
    }

    @Test
    public void utf16() throws IOException {
        // According to the specification 'If charset is a UTF-16 encoding, 
then set charset to UTF-8.'
        assertCharset("<meta charset='UTF-16BE'>", StandardCharsets.UTF_8);
    }

    @Test
    public void macintoshEncoding() throws IOException {
        // In the spec, iso-8859-1 is an alias for WINDOWS-1252
        assertCharset("<meta charset='macintosh'>", 
Charset.forName("x-MacRoman"));
    }

    @Test
    public void iso88591() throws IOException {
        // In the spec, iso-8859-1 is an alias for WINDOWS-1252
        assertWindows1252("<meta charset='iso-8859-1'>");
    }
{code}


was (Author: gbouchar):
I think at least the following tests should be included, and pass:

{{
    @Test
    public void bom() throws IOException {
        // A BOM should have precedence over the meta
        // let assertCharset encode the string in the expected charset
        assertCharset("\ufeff<meta charset='WINDOWS-1252'>", 
StandardCharsets.UTF_8);
        assertCharset("\ufeff<meta charset='WINDOWS-1252'>", 
StandardCharsets.UTF_16LE);
        assertCharset("\ufeff<meta charset='WINDOWS-1252'>", 
StandardCharsets.UTF_16BE);
    }

    @Test
    public void utf16() throws IOException {
        // According to the specification 'If charset is a UTF-16 encoding, 
then set charset to UTF-8.'
        assertCharset("<meta charset='UTF-16BE'>", StandardCharsets.UTF_8);
    }

    @Test
    public void macintoshEncoding() throws IOException {
        // In the spec, iso-8859-1 is an alias for WINDOWS-1252
        assertCharset("<meta charset='macintosh'>", 
Charset.forName("x-MacRoman"));
    }

    @Test
    public void iso88591() throws IOException {
        // In the spec, iso-8859-1 is an alias for WINDOWS-1252
        assertWindows1252("<meta charset='iso-8859-1'>");
    }
}}

> HtmlEncodingDetector doesn't follow the specification
> -----------------------------------------------------
>
>                 Key: TIKA-2673
>                 URL: https://issues.apache.org/jira/browse/TIKA-2673
>             Project: Tika
>          Issue Type: Bug
>            Reporter: Gerard Bouchar
>            Priority: Major
>         Attachments: HtmlEncodingDetectorTest.java
>
>
> This bug is linked to TIKA-2671, but does not concern metadata, but rather 
> the bytes-based detection itself.
> While reading the specification, I collected a list of sample cases where 
> HtmlEncodingDetector differs from the specification, and thus fails at 
> detecting the right charset.
> I am attaching the test cases to this issue: 



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)