[
https://issues.apache.org/jira/browse/TIKA-2673?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16569913#comment-16569913
]
Gerard Bouchar commented on TIKA-2673:
--------------------------------------
[[email protected]], thank you very much for merging my work !
{quote}would you be able to share the urls you used for your study?
{quote}
See [my comment above|#comment-16542763] for the list of URLs I used, together
with the testing methodology, and the code used to analyze the results. The
URL lists are really not big, just a few Mb when compressed. You can [download
the compressed jsonl data
files|https://github.com/GerardBouchar/tika/tree/TIKA-2673-benchmark/benchmark],
and then extract the URLs using jq, for instance:
{code}
< segment_big_icu_charsets.jsonl.xz xz -d | jq -r '.url'
{code}
{quote}Please do check that I got the updates right in both master and
branch_1x (which we'll use for Tika 1.19).{quote}
Everything looks correct to me. 👍
{quote}I made the decision to remove the FullStandardEncodingDetector and the
StandardICU4JEncodingDetector because we try to keep encoding detectors
composable. I was actually wondering if we should pull out the BOM detection
into a separate detector, but let's leave it as is for now.{quote}
OK. I can make a pull request for a separate encoding detector using only the
BOM. But anyway, I think it would be nice and helpful to provide a pre-composed
charset detector that includes all the required step. I think most people want
an encoding detector that "just works" by default.
> HtmlEncodingDetector doesn't follow the specification
> -----------------------------------------------------
>
> Key: TIKA-2673
> URL: https://issues.apache.org/jira/browse/TIKA-2673
> Project: Tika
> Issue Type: Bug
> Reporter: Gerard Bouchar
> Assignee: Tim Allison
> Priority: Major
> Fix For: 1.19, 2.0.0
>
> Attachments: HtmlEncodingDetectorTest.java,
> StrictHtmlEncodingDetector.tar.gz, image-2018-07-13-11-28-16-657.png
>
>
> This bug is linked to TIKA-2671, but does not concern metadata, but rather
> the bytes-based detection itself.
> While reading the specification, I collected a list of sample cases where
> HtmlEncodingDetector differs from the specification, and thus fails at
> detecting the right charset.
> I am attaching the test cases to this issue:
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
