[
https://issues.apache.org/jira/browse/TIKA-2673?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16520281#comment-16520281
]
Gerard Bouchar edited comment on TIKA-2673 at 6/22/18 11:59 AM:
----------------------------------------------------------------
{quote}If you'd like to contribute a StrictHTMLEncodingDetector, we could
compare the performance of that with what we have on our 1TB regression corpus.
{quote}
I agree that testing on real-world data is the key for such a problem. We are
going to conduct our own testing internally, but more testing can only be
beneficial.
I am attaching my first attempt at writing a StrictHtmlEncodingDetector for
tika. The code is still quite messy, but I tried to write a lot of tests, and
have 99% code coverage. It should take into account user-defined metadata,
unicode BOM, and meta tags according to the specification. It is meant to be
used in a composite encoding detector, with an existing probabilistic detector
such as Icu4jEncodingDetector as fallback.
I would be very curious to see how it performs on your corpus, compared to a
real modern browser.
[^StrictHtmlEncodingDetector.tar.gz]
was (Author: gbouchar):
> If you'd like to contribute a StrictHTMLEncodingDetector, we could compare
> the performance of that with what we have on our 1TB regression corpus.
I agree that testing on real-world data is the key for such a problem. We are
going to conduct our own testing internally, but more testing can only be
beneficial.
I am attaching my first attempt at writing a StrictHtmlEncodingDetector for
tika. The code is still quite messy, but I tried to write a lot of tests, and
have 99% code coverage. It should take into account user-defined metadata,
unicode BOM, and meta tags according to the specification. It is meant to be
used in a composite encoding detector, with an existing probabilistic detector
such as Icu4jEncodingDetector as fallback.
I would be very curious to see how it performs on your corpus, compared to a
real modern browser.
[^StrictHtmlEncodingDetector.tar.gz]
> HtmlEncodingDetector doesn't follow the specification
> -----------------------------------------------------
>
> Key: TIKA-2673
> URL: https://issues.apache.org/jira/browse/TIKA-2673
> Project: Tika
> Issue Type: Bug
> Reporter: Gerard Bouchar
> Priority: Major
> Attachments: HtmlEncodingDetectorTest.java,
> StrictHtmlEncodingDetector.tar.gz
>
>
> This bug is linked to TIKA-2671, but does not concern metadata, but rather
> the bytes-based detection itself.
> While reading the specification, I collected a list of sample cases where
> HtmlEncodingDetector differs from the specification, and thus fails at
> detecting the right charset.
> I am attaching the test cases to this issue:
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
