[
https://issues.apache.org/jira/browse/TIKA-2673?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16546162#comment-16546162
]
Gerard Bouchar edited comment on TIKA-2673 at 7/17/18 8:18 AM:
---------------------------------------------------------------
[~wastl-nagel] : I used our internal fork of nutch, with the ICU encoding
detector from tika 1.17 (org.apache.tika.parser.txt.Icu4jEncodingDetector). I
used tika's metadata to set the declared charset (Metadata.CONTENT_ENCODING),
extracting the charset from the HTTP Content-Type header with the following
regex : _charset=\s*"?([^\s;"]*) ._
Our regex is case-sensitive, whereas it should not, according to the spec. But
I had to test it that way because this is how our internal crawler worked, and
I wanted to compare new results to our current baseline.
By the way, the HTTP header _Content-Encoding_ used by Icu4jEncodingDetector as
a character set declaration, is in fact a compression algorithm declaration
according to [the
specification|https://tools.ietf.org/html/rfc7231#section-3.1.2.2].
was (Author: gbouchar):
[~wastl-nagel] : I used our internal fork of nutch, with the ICU encoding
detector from tika 1.17 (org.apache.tika.parser.txt.Icu4jEncodingDetector). I
used tika's metadata to set the declared charset (Metadata.CONTENT_ENCODING),
extracting the charset from the HTTP Content-Type header with the following
regex : _charset=\s*"?([^\s;"]*) ._
By the way, this is a strange way to do, as the HTTP header Content-Encoding is
not meant to declare a charset, but a compression algorithm. And our regex is
case-sensitive, whereas it should not, according to the spec. But I had to test
it that way because this is how our internal crawler worked, and I wanted to
compare new results to our current baseline.
> HtmlEncodingDetector doesn't follow the specification
> -----------------------------------------------------
>
> Key: TIKA-2673
> URL: https://issues.apache.org/jira/browse/TIKA-2673
> Project: Tika
> Issue Type: Bug
> Reporter: Gerard Bouchar
> Priority: Major
> Attachments: HtmlEncodingDetectorTest.java,
> StrictHtmlEncodingDetector.tar.gz, image-2018-07-13-11-28-16-657.png
>
>
> This bug is linked to TIKA-2671, but does not concern metadata, but rather
> the bytes-based detection itself.
> While reading the specification, I collected a list of sample cases where
> HtmlEncodingDetector differs from the specification, and thus fails at
> detecting the right charset.
> I am attaching the test cases to this issue:
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
