[
https://issues.apache.org/jira/browse/TIKA-5052?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=18139788#comment-18139788
]
Tim Allison commented on TIKA-5052:
-----------------------------------
Final evaluation on the held out "dev" set. I'm keeping the held out held out
test set for later.
||Dev set||n||main (4.3.0-SNAPSHOT)||4.2.0||3.4.0-SNAPSHOT||
|cc-html|1,922|*99.64*|97.24|87.30|
|targeted|99|*97.98*|79.80|90.91|
|aug1|2,051|*98.68*|94.83|86.88|
|aug2|153|*97.39*|75.82|79.74|
|CC v2|3,889|*97.25*|94.91|91.90|
|CC v3 A1|226|*99.12*|98.23|82.74|
|CC v3 A2|278|*99.28*|96.40|95.68|
|CC v3 B1|1,125|97.42|*99.20*|90.93|
|CC v3 B2|654|*94.34*|82.42|81.04|
|CC v3 pooled|2,283|*96.93*|93.96|87.87|
|UTF-16|23,040|*99.94*|84.53|63.48|
Datasets:
* *cc-html*: Common Crawl HTML pages that contain non-ASCII bytes, with
hand-adjudicated charset labels.
* *targeted*: cc-html pages chosen for rare or confusable charsets
(ISO-8859-5/6/7/8, windows-1253), labelled from the bytes that decide between
them.
* *aug1*: a random sample (with known sampling weights) of further cc-html
pages, added to cover more charsets.
* *aug2*: a second, smaller augmentation batch of cc-html pages.
* *CC v2*: non-HTML text from Common Crawl (email, plain text, CSV, subtitles
and other text types).
* *CC v3 A1*: non-HTML Common Crawl text from Greek, Turkish, Hebrew and Arabic
sites.
* *CC v3 A2*: non-HTML Common Crawl text from Cyrillic, Central European,
Baltic and Korean sites.
* *CC v3 B1*: non-HTML Common Crawl text from Thai, Vietnamese, Japanese and
Chinese sites.
* *CC v3 B2*: non-HTML Common Crawl text from Western European sites, plus .nfo
files, subtitles and DOS-era readmes from anywhere.
* *UTF-16*: synthetic UTF-16LE/BE text without a BOM, made from held-out
Wikipedia text in many scripts.
> Update encoding detector
> ------------------------
>
> Key: TIKA-5052
> URL: https://issues.apache.org/jira/browse/TIKA-5052
> Project: Tika
> Issue Type: Task
> Reporter: Tim Allison
> Priority: Major
>
> The recent Arabic ISO issue encouraged me to look again at mojibuster. I've
> spent the last few days curating and improving a train/dev/test corpus
> derived largely from [~jnioche]'s common-crawl html contribution back in
> January 2024(?). The corpus is heavily oversampled for non-ascii and is a
> treasure trove.
> Along the way, I experimented with a hashed byte ngram model, and it seems to
> work much better so far than the earlier naive bayes approach. This model
> also incorporates UTF16 rather than adjudicating a side car wide unicode
> classifier.
> I'll continue to have bots tagging the original html corpus and supplement it
> with other datasets to make training and evaluating new models easier.
> Initial results with a cnn model were promising, but a bit slow without jdk25
> vectorization. I'll likely keep working on that.
> And, of course, we can add more charsets as needed: DOS and ebcdic maybe?
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
