[
https://issues.apache.org/jira/browse/TIKA-5019?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=18137296#comment-18137296
]
Tim Allison commented on TIKA-5019:
-----------------------------------
To be clear, the 76.9% in 3.x is mostly caused by an html header declaring the
narrow charset and then Tika decoding it as such, rather than backing off to
the super charset for decoding (e.g. ).
This more accurately reflects what's going on in 3x vs 4x.
||Version||Strict accuracy||Accuracy if 3.x read the header like
browsers||Pages with >50 chars wrong||
|3.x (branch_3x)|76.9%|≈93.5%|87|
|4.2.0|97.2%|—|15|
|4.x with this change|*98.3%*|—|8|
> Detecting Arabic encoded with ISO-8859-6 no longer works
> --------------------------------------------------------
>
> Key: TIKA-5019
> URL: https://issues.apache.org/jira/browse/TIKA-5019
> Project: Tika
> Issue Type: Bug
> Components: detector
> Affects Versions: 4.2.0
> Reporter: Boris Petrov
> Priority: Major
>
> I'm upgrading from Tika 3 and this worked fine:
> Groovy code:
> createAndTestFile('جديد للوقاية من العنف وإساءة معاملة الأطفال.',
> Charset.forName('ISO-8859-6')) == 'ISO-8859-6'
> This creates a test file with this text using the specified encoding and then
> uses Tika's DefaultEncodingDetector to detect the charset on it. I explicitly
> used to remove the `UniversalEncodingDetector` because it didn't work OK.
> Otherwise this (and other cases) worked fine.
> With Tika 4 some test cases changed, most are fine but this particularly
> isn't. It returns `x-EUC-TW` instead as the encoding. If I "multiply" the
> string by 20 (that is, copy-paste this text 20 times), then the detector
> returns `windows-1252` - which is also wrong.
> Let me know if you need more information or somehow can't reproduce what I'm
> saying.
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
