tballison opened a new pull request, #3550:
URL: https://github.com/apache/tika/pull/3550
…A-5048 UTF-16/UTF-32 fixes
From the bot:
```
MojibusterEncodingDetector's statistical stage is a hashed byte n-gram model
(HNG2, int8 tables, even/odd column channels, mean+max pooling, one hidden
layer), trained on gold v1.12, with UTF-16LE/BE, UTF-32LE/BE and binary
classes. Runner-ups within 1.0 logit become candidates (calibrated on the
full
chain). The UTF-16 specialist, its gate and the shortcut that hid UTF-32
are
removed; a surrogate pair cut at the probe end no longer vetoes UTF-16; C0
controls no longer pass as pure ASCII. Scoring is JIT-friendly and stops
early
on long probes once the top class leads by 2 logits (dense-script latency
within ~1.5x of the old chain). Removes NaiveBayesBigramEncodingDetector,
nb-bigram.bin and its trainer.
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
