[
https://issues.apache.org/jira/browse/TIKA-5052?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=18141692#comment-18141692
]
Hudson commented on TIKA-5052:
------------------------------
SUCCESS: Integrated in Jenkins build Tika ยป tika-main-jdk17 #1791 (See
[https://ci-builds.apache.org/job/Tika/job/tika-main-jdk17/1791/])
TIKA-5052: Big5 whatwg (#3595) (github:
[https://github.com/apache/tika/commit/a40d54f6a6d9cf8ce485f10bb07011cd32ef5925])
* (edit)
tika-server/tika-server-standard/src/main/appended-resources/META-INF/LICENSE
* (edit)
tika-core/src/test/java/org/apache/tika/detect/CharsetSupersetsTest.java
* (edit)
tika-bundles/tika-bundle-standard/src/main/appended-resources/META-INF/LICENSE
* (add) tika-core/src/test/java/org/apache/tika/detect/WhatwgPatchGenerator.java
* (edit) tika-core/src/main/java/org/apache/tika/detect/AutoDetectReader.java
* (add)
tika-core/src/main/resources/org/apache/tika/detect/x-eucJP-Open-WHATWG.tsv
* (edit) LICENSE.txt
* (edit)
tika-parsers/tika-parsers-standard/tika-parsers-standard-modules/tika-parser-microsoft-module/src/main/java/org/apache/tika/parser/microsoft/OutlookExtractor.java
* (edit) tika-app/src/main/appended-resources/META-INF/LICENSE
* (edit)
tika-ml/tika-ml-junkdetect/src/main/java/org/apache/tika/ml/junkdetect/JunkFilterEncodingDetector.java
* (add)
tika-core/src/main/resources/org/apache/tika/detect/x-Big5-HKSCS-WHATWG.tsv
* (add)
tika-core/src/test/java/org/apache/tika/detect/DecodeAsNameUsageTest.java
* (edit) CHANGES.txt
* (edit)
tika-parsers/tika-parsers-standard/tika-parsers-standard-modules/tika-parser-html-module/src/test/java/org/apache/tika/parser/html/HtmlParserTest.java
* (edit) tika-core/src/main/appended-resources/META-INF/LICENSE
* (edit)
tika-parsers/tika-parsers-standard/tika-parsers-standard-modules/tika-parser-html-module/src/main/java/org/apache/tika/parser/html/JSoupParser.java
* (edit) docs/modules/ROOT/pages/advanced/charset-detection-design.adoc
* (add) tika-core/src/main/java/org/apache/tika/detect/PatchedCharset.java
* (edit) tika-core/src/main/java/org/apache/tika/detect/CharsetSupersets.java
* (edit) tika-core/pom.xml
* (add) tika-core/src/test/java/org/apache/tika/detect/PatchedCharsetTest.java
> Update encoding detector
> ------------------------
>
> Key: TIKA-5052
> URL: https://issues.apache.org/jira/browse/TIKA-5052
> Project: Tika
> Issue Type: Task
> Reporter: Tim Allison
> Priority: Major
>
> The recent Arabic ISO issue encouraged me to look again at mojibuster. I've
> spent the last few days curating and improving a train/dev/test corpus
> derived largely from [~jnioche]'s common-crawl html contribution back in
> January 2024(?). The corpus is heavily oversampled for non-ascii and is a
> treasure trove.
> Along the way, I experimented with a hashed byte ngram model, and it seems to
> work much better so far than the earlier naive bayes approach. This model
> also incorporates UTF16 rather than adjudicating a side car wide unicode
> classifier.
> I'll continue to have bots tagging the original html corpus and supplement it
> with other datasets to make training and evaluating new models easier.
> Initial results with a cnn model were promising, but a bit slow without jdk25
> vectorization. I'll likely keep working on that.
> And, of course, we can add more charsets as needed: DOS and ebcdic maybe?
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
