[
https://issues.apache.org/jira/browse/TIKA-2673?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16545135#comment-16545135
]
Sebastian Nagel commented on TIKA-2673:
---------------------------------------
Hi [~gbouchar], one question: did you evaluate Tika's modified version of ICU's
encoding detector or the original version? And did you call
setDeclaredEncoding(...)? Background: ICU's charset detector does not take the
"declared" encoding (from HTTP or HTML metadata) into account - the
implementation contradicts the documentation
([setDeclaredEncoding(...)|http://icu-project.org/apiref/icu4j/com/ibm/icu/text/CharsetDetector.html#setDeclaredEncoding-java.lang.String-]
vs.
[CharsetDetector.java|http://source.icu-project.org/repos/icu/trunk/icu4j/main/classes/core/src/com/ibm/icu/text/CharsetDetector.java]).
Tika uses a [patched
version|https://github.com/apache/tika/blob/master/tika-parsers/src/main/java/org/apache/tika/parser/txt/CharsetDetector.java#L320]
(TIKA-335) of the ICU detector which boosts the declared encoding if also
detected as possible encoding. That's very important for single-byte encodings,
e.g, of the ISO-8859-x family. Also note: Nutch uses the ICU version (and also
does not set the declared encoding).
> HtmlEncodingDetector doesn't follow the specification
> -----------------------------------------------------
>
> Key: TIKA-2673
> URL: https://issues.apache.org/jira/browse/TIKA-2673
> Project: Tika
> Issue Type: Bug
> Reporter: Gerard Bouchar
> Priority: Major
> Attachments: HtmlEncodingDetectorTest.java,
> StrictHtmlEncodingDetector.tar.gz, image-2018-07-13-11-28-16-657.png
>
>
> This bug is linked to TIKA-2671, but does not concern metadata, but rather
> the bytes-based detection itself.
> While reading the specification, I collected a list of sample cases where
> HtmlEncodingDetector differs from the specification, and thus fails at
> detecting the right charset.
> I am attaching the test cases to this issue:
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
