[ https://issues.apache.org/jira/browse/TIKA-2673?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16520281#comment-16520281 ]
Gerard Bouchar commented on TIKA-2673: -------------------------------------- > If you'd like to contribute a StrictHTMLEncodingDetector, we could compare > the performance of that with what we have on our 1TB regression corpus. I agree that testing on real-world data is the key for such a problem. We are going to conduct our own testing internally, but more testing can only be beneficial. I am attaching my first attempt at writing a StrictHtmlEncodingDetector for tika. The code is still quite messy, but I tried to write a lot of tests, and have 99% code coverage. It should take into account user-defined metadata, unicode BOM, and meta tags according to the specification. It is meant to be used in a composite encoding detector, with an existing probabilistic detector such as Icu4jEncodingDetector as fallback. I would be very curious to see how it performs on your corpus, compared to a real modern browser. [^StrictHtmlEncodingDetector.tar.gz] > HtmlEncodingDetector doesn't follow the specification > ----------------------------------------------------- > > Key: TIKA-2673 > URL: https://issues.apache.org/jira/browse/TIKA-2673 > Project: Tika > Issue Type: Bug > Reporter: Gerard Bouchar > Priority: Major > Attachments: HtmlEncodingDetectorTest.java, > StrictHtmlEncodingDetector.tar.gz > > > This bug is linked to TIKA-2671, but does not concern metadata, but rather > the bytes-based detection itself. > While reading the specification, I collected a list of sample cases where > HtmlEncodingDetector differs from the specification, and thus fails at > detecting the right charset. > I am attaching the test cases to this issue: -- This message was sent by Atlassian JIRA (v7.6.3#76005)