[
https://issues.apache.org/jira/browse/TIKA-5019?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=18138891#comment-18138891
]
Hudson commented on TIKA-5019:
------------------------------
UNSTABLE: Integrated in Jenkins build Tika » tika-main-jdk17 #1775 (See
[https://ci-builds.apache.org/job/Tika/job/tika-main-jdk17/1775/])
TIKA-5019: detect undeclared ISO-8859-5/6/7 (#3521) (github:
[https://github.com/apache/tika/commit/af0e790e2da85ae894ec7cc56541f7a695a6044f])
* (edit) tika-core/src/main/java/org/apache/tika/detect/EncodingResult.java
* (edit)
tika-ml/tika-ml-chardetect/src/test/java/org/apache/tika/ml/chardetect/CharsetConfusablesTest.java
* (edit)
tika-ml/tika-ml-chardetect/src/main/java/org/apache/tika/ml/chardetect/tools/TrainNaiveBayesBigram.java
* (edit) tika-core/src/test/java/org/apache/tika/detect/BOMDetectorTest.java
* (edit)
tika-encoding-detectors/tika-encoding-detector-mojibuster/src/main/java/org/apache/tika/ml/chardetect/CharsetConfusables.java
* (edit)
tika-ml/tika-ml-chardetect/src/main/java/org/apache/tika/ml/chardetect/tools/BuildCharsetTrainingData.java
* (edit) tika-ml/tika-ml-chardetect/pom.xml
* (edit) tika-core/src/main/java/org/apache/tika/detect/AutoDetectReader.java
* (add)
tika-ml/tika-ml-chardetect/src/main/java/org/apache/tika/ml/chardetect/tools/WikiSplits.java
* (add)
tika-ml/tika-ml-junkdetect/src/test/java/org/apache/tika/ml/junkdetect/Utf16BomDetectionTest.java
* (add)
tika-ml/tika-ml-junkdetect/src/main/resources/org/apache/tika/ml/junkdetect/legacy-charset-frequency.txt
* (edit)
tika-ml/tika-ml-junkdetect/src/main/java/org/apache/tika/ml/junkdetect/JunkFilterEncodingDetector.java
* (add)
tika-ml/tika-ml-junkdetect/src/test/java/org/apache/tika/ml/junkdetect/IdenticalDecodingsTest.java
* (edit)
tika-parsers/tika-parsers-standard/tika-parsers-standard-modules/tika-parser-html-module/src/test/java/org/apache/tika/parser/html/HtmlParserTest.java
* (edit) tika-core/src/main/java/org/apache/tika/detect/BOMDetector.java
* (add)
tika-ml/tika-ml-chardetect/src/main/java/org/apache/tika/ml/chardetect/tools/CharsetTextEncoding.java
* (edit) CHANGES.txt
* (add)
tika-ml/tika-ml-chardetect/src/main/java/org/apache/tika/ml/chardetect/tools/CompareCharsetEvalRuns.java
* (edit)
tika-ml/tika-ml-junkdetect/src/test/java/org/apache/tika/ml/junkdetect/JunkFilterEncodingDetectorTest.java
* (add)
tika-ml/tika-ml-junkdetect/src/test/java/org/apache/tika/ml/junkdetect/IsoCharsetDetectionTest.java
* (edit)
tika-parsers/tika-parsers-standard/tika-parsers-standard-modules/tika-parser-html-module/src/main/java/org/apache/tika/parser/html/JSoupParser.java
* (edit)
tika-encoding-detectors/tika-encoding-detector-mojibuster/src/main/java/org/apache/tika/ml/chardetect/MojibusterEncodingDetector.java
* (add)
tika-ml/tika-ml-chardetect/src/main/java/org/apache/tika/ml/chardetect/tools/BuildCharsetEvalData.java
* (add)
tika-ml/tika-ml-junkdetect/src/test/java/org/apache/tika/ml/junkdetect/RejectedBomReaderTest.java
* (edit)
tika-encoding-detectors/tika-encoding-detector-mojibuster/src/main/resources/org/apache/tika/ml/chardetect/nb-bigram.bin
* (edit)
tika-encoding-detectors/tika-encoding-detector-mojibuster/src/main/java/org/apache/tika/ml/chardetect/NaiveBayesBigramEncodingDetector.java
* (edit) tika-core/src/main/java/org/apache/tika/detect/EncodingDetector.java
* (add)
tika-ml/tika-ml-chardetect/src/main/java/org/apache/tika/ml/chardetect/tools/ScoreCharsetEvalData.java
> Detecting Arabic encoded with ISO-8859-6 no longer works
> --------------------------------------------------------
>
> Key: TIKA-5019
> URL: https://issues.apache.org/jira/browse/TIKA-5019
> Project: Tika
> Issue Type: Bug
> Components: detector
> Affects Versions: 4.2.0
> Reporter: Boris Petrov
> Priority: Major
>
> I'm upgrading from Tika 3 and this worked fine:
> Groovy code:
> createAndTestFile('جديد للوقاية من العنف وإساءة معاملة الأطفال.',
> Charset.forName('ISO-8859-6')) == 'ISO-8859-6'
> This creates a test file with this text using the specified encoding and then
> uses Tika's DefaultEncodingDetector to detect the charset on it. I explicitly
> used to remove the `UniversalEncodingDetector` because it didn't work OK.
> Otherwise this (and other cases) worked fine.
> With Tika 4 some test cases changed, most are fine but this particularly
> isn't. It returns `x-EUC-TW` instead as the encoding. If I "multiply" the
> string by 20 (that is, copy-paste this text 20 times), then the detector
> returns `windows-1252` - which is also wrong.
> Let me know if you need more information or somehow can't reproduce what I'm
> saying.
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
