[
https://issues.apache.org/jira/browse/TIKA-5019?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=18137183#comment-18137183
]
Tim Allison commented on TIKA-5019:
-----------------------------------
General: cc-html dev, 1,933 pages, natural mix
h4. General web pages (cc-html dev, 1,933 pages, natural charset mix)
||Version||Accuracy||Errors||ms/doc (3 runs)||KB allocated/doc||
|3.x (branch_3x)|76.9%|446|0.054-0.055|94|
|4.2.0|97.2%|54|1.36-1.49|1,287|
|4.x with this change|*98.3%*|33|1.30-1.50|1,319|
Targeted: ISO-8859-5/6/7/8 and windows-1253 pages
h4. Targeted pages (cc-html ISO-8859-5/6/7/8 + windows-1253 stratum)
||Version||Accuracy (dev, 75 pages)||ms/doc (528 pages, 3 runs)||KB
allocated/doc||
|3.x (branch_3x)|89.3%|0.070-0.079|88|
|4.2.0|84.0%|1.66-1.75|1,284|
|4.x with this change|*94.7%*|2.05-2.20|1,614|
Single-threaded, whole file, no metadata hints; best of 3 passes after
warm-up. A prediction counts as correct when it decodes the page identically to
the gold label.
> Detecting Arabic encoded with ISO-8859-6 no longer works
> --------------------------------------------------------
>
> Key: TIKA-5019
> URL: https://issues.apache.org/jira/browse/TIKA-5019
> Project: Tika
> Issue Type: Bug
> Components: detector
> Affects Versions: 4.2.0
> Reporter: Boris Petrov
> Priority: Major
>
> I'm upgrading from Tika 3 and this worked fine:
> Groovy code:
> createAndTestFile('جديد للوقاية من العنف وإساءة معاملة الأطفال.',
> Charset.forName('ISO-8859-6')) == 'ISO-8859-6'
> This creates a test file with this text using the specified encoding and then
> uses Tika's DefaultEncodingDetector to detect the charset on it. I explicitly
> used to remove the `UniversalEncodingDetector` because it didn't work OK.
> Otherwise this (and other cases) worked fine.
> With Tika 4 some test cases changed, most are fine but this particularly
> isn't. It returns `x-EUC-TW` instead as the encoding. If I "multiply" the
> string by 20 (that is, copy-paste this text 20 times), then the detector
> returns `windows-1252` - which is also wrong.
> Let me know if you need more information or somehow can't reproduce what I'm
> saying.
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
