[
https://issues.apache.org/jira/browse/TIKA-5048?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=18139201#comment-18139201
]
Hudson commented on TIKA-5048:
------------------------------
UNSTABLE: Integrated in Jenkins build Tika ยป tika-main-jdk17 #1778 (See
[https://ci-builds.apache.org/job/Tika/job/tika-main-jdk17/1778/])
TIKA-5052: hashed byte n-gram charset model replaces Naive Bayes; TIKA-5048
UTF-16/UTF-32 fixes (#3550) (github:
[https://github.com/apache/tika/commit/fbcf973274b3998f85009152257049de178a6b6d])
* (edit)
tika-encoding-detectors/tika-encoding-detector-mojibuster/src/main/java/org/apache/tika/ml/chardetect/CharsetConfusables.java
* (delete)
tika-encoding-detectors/tika-encoding-detector-mojibuster/src/main/resources/META-INF/services/org.apache.tika.ml.chardetect.StatisticalSpecialist
* (edit) docs/modules/ROOT/pages/configuration/encoding-detectors.adoc
* (edit)
tika-ml/tika-ml-chardetect/src/main/java/org/apache/tika/ml/chardetect/tools/EvalCharsetDetectors.java
* (add)
tika-encoding-detectors/tika-encoding-detector-mojibuster/src/main/java/org/apache/tika/ml/chardetect/HashedNgramEncodingDetector.java
* (delete)
tika-encoding-detectors/tika-encoding-detector-mojibuster/src/test/java/org/apache/tika/ml/chardetect/Utf16SpecialistEncodingDetectorTestFixtures.java
* (delete)
tika-encoding-detectors/tika-encoding-detector-mojibuster/src/main/resources/org/apache/tika/ml/chardetect/utf16-specialist.bin
* (delete)
tika-encoding-detectors/tika-encoding-detector-mojibuster/src/test/java/org/apache/tika/ml/chardetect/CalibrateTopK.java
* (edit) CHANGES.txt
* (edit) tika-ml/tika-ml-chardetect/pom.xml
* (delete)
tika-encoding-detectors/tika-encoding-detector-mojibuster/src/main/java/org/apache/tika/ml/chardetect/SpecialistOutput.java
* (delete)
tika-encoding-detectors/tika-encoding-detector-mojibuster/src/main/java/org/apache/tika/ml/chardetect/StatisticalSpecialist.java
* (delete)
tika-ml/tika-ml-chardetect/src/main/java/org/apache/tika/ml/chardetect/tools/TrainNaiveBayesBigram.java
* (delete)
tika-encoding-detectors/tika-encoding-detector-mojibuster/src/test/java/org/apache/tika/ml/chardetect/InspectBigramContributions.java
* (edit) tika-encoding-detectors/tika-encoding-detector-mojibuster/pom.xml
* (add)
tika-encoding-detectors/tika-encoding-detector-mojibuster/src/main/java/org/apache/tika/ml/chardetect/HashedNgramModel.java
* (delete)
tika-encoding-detectors/tika-encoding-detector-mojibuster/src/test/java/org/apache/tika/ml/chardetect/Utf16ColumnFeatureExtractorTest.java
* (add)
tika-encoding-detectors/tika-encoding-detector-mojibuster/src/test/java/org/apache/tika/ml/chardetect/HashedNgramModelParityTest.java
* (delete)
tika-encoding-detectors/tika-encoding-detector-mojibuster/src/main/java/org/apache/tika/ml/chardetect/NaiveBayesBigramEncodingDetector.java
* (add)
tika-encoding-detectors/tika-encoding-detector-mojibuster/src/test/resources/org/apache/tika/ml/chardetect/hashed-ngram-parity.tsv
* (edit)
tika-encoding-detectors/tika-encoding-detector-mojibuster/src/test/java/org/apache/tika/ml/chardetect/ModelResourceUniquenessTest.java
* (delete)
tika-encoding-detectors/tika-encoding-detector-mojibuster/src/main/java/org/apache/tika/ml/chardetect/Utf16SpecialistEncodingDetector.java
* (add)
tika-encoding-detectors/tika-encoding-detector-mojibuster/src/test/java/org/apache/tika/ml/chardetect/ShortUtf8ProbeTest.java
* (edit)
tika-encoding-detectors/tika-encoding-detector-mojibuster/src/test/java/org/apache/tika/ml/chardetect/TraceMojibuster.java
* (add)
tika-encoding-detectors/tika-encoding-detector-mojibuster/src/test/java/org/apache/tika/ml/chardetect/HashedNgramEarlyStopTest.java
* (add)
tika-encoding-detectors/tika-encoding-detector-mojibuster/src/test/java/org/apache/tika/ml/chardetect/WideUnicodeDetectionTest.java
* (edit)
tika-encoding-detectors/tika-encoding-detector-mojibuster/src/main/java/org/apache/tika/ml/chardetect/MojibusterEncodingDetector.java
* (edit) docs/modules/ROOT/pages/advanced/charset-detection-design.adoc
* (edit) tika-ml/tika-ml-chardetect/README.md
* (edit)
tika-encoding-detectors/tika-encoding-detector-mojibuster/src/main/java/org/apache/tika/ml/chardetect/StructuralEncodingRules.java
* (delete)
tika-encoding-detectors/tika-encoding-detector-mojibuster/src/test/java/org/apache/tika/ml/chardetect/Utf16SpecialistEncodingDetectorTest.java
* (edit)
tika-ml/tika-ml-junkdetect/src/test/java/org/apache/tika/ml/junkdetect/Utf16BomDetectionTest.java
* (edit)
tika-encoding-detectors/tika-encoding-detector-mojibuster/src/main/java/org/apache/tika/ml/chardetect/WideUnicodeDetector.java
* (edit)
tika-ml/tika-ml-chardetect/src/main/java/org/apache/tika/ml/chardetect/tools/BuildCharsetTrainingData.java
* (edit)
tika-ml/tika-ml-chardetect/src/main/java/org/apache/tika/ml/chardetect/tools/RebalanceCharsetTraining.java
* (delete)
tika-encoding-detectors/tika-encoding-detector-mojibuster/src/main/resources/org/apache/tika/ml/chardetect/nb-bigram.bin
* (add)
tika-encoding-detectors/tika-encoding-detector-mojibuster/src/main/resources/org/apache/tika/ml/chardetect/hashed-ngram.hng
* (delete)
tika-encoding-detectors/tika-encoding-detector-mojibuster/src/main/java/org/apache/tika/ml/chardetect/Utf16ColumnFeatureExtractor.java
* (add)
tika-encoding-detectors/tika-encoding-detector-mojibuster/src/main/resources/org/apache/tika/ml/chardetect/hashed-ngram.hng.provenance
> UTF-16/UTF-32 detection: one emoji defeats UTF-16 without a BOM, UTF-32 is
> reported as UTF-16
> ---------------------------------------------------------------------------------------------
>
> Key: TIKA-5048
> URL: https://issues.apache.org/jira/browse/TIKA-5048
> Project: Tika
> Issue Type: Task
> Reporter: Tim Allison
> Priority: Major
>
> and a surrogate pair cut at the probe edge vetoes a correct call.
> Bot found 3 areas for improvement in utf16. We'll likely fix these as part of
> a larger charset encoding detection fix.
> I wanted to document these limitations.
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
