[
https://issues.apache.org/jira/browse/TIKA-5052?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=18138882#comment-18138882
]
ASF GitHub Bot commented on TIKA-5052:
--------------------------------------
tballison opened a new pull request, #3550:
URL: https://github.com/apache/tika/pull/3550
…A-5048 UTF-16/UTF-32 fixes
From the bot:
```
MojibusterEncodingDetector's statistical stage is a hashed byte n-gram model
(HNG2, int8 tables, even/odd column channels, mean+max pooling, one hidden
layer), trained on gold v1.12, with UTF-16LE/BE, UTF-32LE/BE and binary
classes. Runner-ups within 1.0 logit become candidates (calibrated on the
full
chain). The UTF-16 specialist, its gate and the shortcut that hid UTF-32
are
removed; a surrogate pair cut at the probe end no longer vetoes UTF-16; C0
controls no longer pass as pure ASCII. Scoring is JIT-friendly and stops
early
on long probes once the top class leads by 2 logits (dense-script latency
within ~1.5x of the old chain). Removes NaiveBayesBigramEncodingDetector,
nb-bigram.bin and its trainer.
```
> Update encoding detector
> ------------------------
>
> Key: TIKA-5052
> URL: https://issues.apache.org/jira/browse/TIKA-5052
> Project: Tika
> Issue Type: Task
> Reporter: Tim Allison
> Priority: Major
>
> The recent Arabic ISO issue encouraged me to look again at mojibuster. I've
> spent the last few days curating and improving a train/dev/test corpus
> derived largely from [~jnioche]'s common-crawl html contribution back in
> January 2024(?). The corpus is heavily oversampled for non-ascii and is a
> treasure trove.
> Along the way, I experimented with a hashed byte ngram model, and it seems to
> work much better so far than the earlier naive bayes approach. This model
> also incorporates UTF16 rather than adjudicating a side car wide unicode
> classifier.
> I'll continue to have bots tagging the original html corpus and supplement it
> with other datasets to make training and evaluating new models easier.
> Initial results with a cnn model were promising, but a bit slow without jdk25
> vectorization. I'll likely keep working on that.
> And, of course, we can add more charsets as needed: DOS and ebcdic maybe?
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
