[
https://issues.apache.org/jira/browse/TIKA-5027?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=18144093#comment-18144093
]
Hudson commented on TIKA-5027:
------------------------------
SUCCESS: Integrated in Jenkins build Tika ยป tika-main-jdk17 #1793 (See
[https://ci-builds.apache.org/job/Tika/job/tika-main-jdk17/1793/])
TIKA-5027: junk utf16 misreads (#3602) (github:
[https://github.com/apache/tika/commit/1c2a4d7f3b6534c9ce440b45cf6ae8e79709b4fc])
* (edit)
tika-ml/tika-ml-junkdetect/src/test/java/org/apache/tika/ml/junkdetect/HashedArbitrationTest.java
* (edit) CHANGES.txt
* (edit)
tika-encoding-detectors/tika-encoding-detector-html/src/main/java/org/apache/tika/detect/html/HtmlEncodingDetector.java
* (edit) tika-ml/tika-ml-junkdetect-tools/train-junk-model.sh
* (edit)
tika-ml/tika-ml-junkdetect/src/main/resources/org/apache/tika/ml/junkdetect/junkdetect.bin.provenance
* (edit)
tika-encoding-detectors/tika-encoding-detector-html/src/test/java/org/apache/tika/parser/html/HtmlEncodingDetectorTest.java
* (edit)
tika-encoding-detectors/tika-encoding-detector-mojibuster/src/test/java/org/apache/tika/ml/chardetect/WideUnicodeDetectionTest.java
* (edit)
tika-parsers/tika-parsers-standard/tika-parsers-standard-integration-tests/src/test/java/org/apache/tika/parser/html/HtmlEncodingDetectionTest.java
* (edit)
tika-encoding-detectors/tika-encoding-detector-mojibuster/src/main/resources/org/apache/tika/ml/chardetect/hashed-ngram.hng.provenance
* (edit)
tika-encoding-detectors/tika-encoding-detector-mojibuster/src/main/resources/org/apache/tika/ml/chardetect/hashed-ngram.hng
* (edit)
tika-ml/tika-ml-junkdetect-tools/src/main/java/org/apache/tika/ml/junkdetect/tools/JunkFamilies.java
* (edit)
tika-encoding-detectors/tika-encoding-detector-mojibuster/src/test/resources/org/apache/tika/ml/chardetect/hashed-ngram-parity.tsv
* (edit)
tika-ml/tika-ml-junkdetect/src/main/resources/org/apache/tika/ml/junkdetect/junkdetect.bin
* (add)
tika-ml/tika-ml-junkdetect/src/test/java/org/apache/tika/ml/junkdetect/LoneWideArbitrationTest.java
* (edit)
tika-encoding-detectors/tika-encoding-detector-mojibuster/src/main/java/org/apache/tika/ml/chardetect/MojibusterEncodingDetector.java
> Improve junk detector
> ---------------------
>
> Key: TIKA-5027
> URL: https://issues.apache.org/jira/browse/TIKA-5027
> Project: Tika
> Issue Type: Task
> Reporter: Tim Allison
> Priority: Major
> Fix For: 4.3.0
>
>
> In looking at charset detection, I found a number of things we could improve
> in junk detection.
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
