Tim Allison created TIKA-5052:
---------------------------------
Summary: Update encoding detector
Key: TIKA-5052
URL: https://issues.apache.org/jira/browse/TIKA-5052
Project: Tika
Issue Type: Task
Reporter: Tim Allison
The recent Arabic ISO issue encouraged me to look again at mojibuster. I've
spent the last few days curating and improving a train/dev/test corpus derived
largely from [~jnioche]'s common-crawl html contribution back in January
2024(?). The corpus is heavily oversampled for non-ascii and is a treasure
trove.
Along the way, I experimented with a hashed byte ngram model, and it seems to
work much better so far than the earlier naive bayes approach. This model also
incorporates UTF16 rather than adjudicating a side car wide unicode classifier.
I'll continue to have bots tagging the original html corpus and supplement it
with other datasets to make training and evaluating new models easier.
Initial results with a cnn model were promising, but a bit slow without jdk25
vectorization. I'll likely keep working on that.
And, of course, we can add more charsets as needed: DOS and ebcdic maybe?
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
