Boris Petrov created TIKA-5019:
----------------------------------
Summary: Detecting Arabic encoded with ISO-8859-6 no longer works
Key: TIKA-5019
URL: https://issues.apache.org/jira/browse/TIKA-5019
Project: Tika
Issue Type: Bug
Components: detector
Affects Versions: 4.2.0
Reporter: Boris Petrov
I'm upgrading from Tika 3 and this worked fine:
Groovy code:
createAndTestFile('جديد للوقاية من العنف وإساءة معاملة الأطفال.',
Charset.forName('ISO-8859-6')) == 'ISO-8859-6'
This creates a test file with this text using the specified encoding and then
uses Tika's DefaultEncodingDetector to detect the charset on it. I explicitly
used to remove the `UniversalEncodingDetector` because it didn't work OK.
Otherwise this (and other cases) worked fine.
With Tika 4 some test cases changed, most are fine but this particularly isn't.
It returns `x-EUC-TW` instead as the encoding. If I "multiply" the string by 20
(that is, copy-paste this text 20 times), then the detector returns
`windows-1252` - which is also wrong.
Let me know if you need more information or somehow can't reproduce what I'm
saying.
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
