https://bugs.documentfoundation.org/show_bug.cgi?id=172905
--- Comment #14 from Julien Nabet <[email protected]> --- Digging a bit more, obviously the way of detection is different between character sets but those like iso-8859-... start with a confidence of 0 and ICU tries to find "prooves" it corresponds to one of them whereas character sets like UTF-16LE start confidence from 10 and then it's adjusted. Since numbers and punctuations are ignored to detect iso-8859, the rest of the file has too few letters (accentuated or not) to increase confidence for iso-8859. Badfully, it means that a lot of statistics files (at least those containing ° encoded with B0 so some weather stats for example or any physics stats which include temperature not encoded in UTF-8) may be wrongly detected with Chinese glyphs. So I still think the patch quoted in my previous comment ( + if (ucsdet_getConfidence(match, &uerr) <= 10) + return; ) may be relevant. We could also add a section in option to enable/disable character sets detected by default. It would require some UI change + use of lib icu function "ucsdet_detectAll" to filter out the non desired character sets. (I've submitted a patch to display character sets detected by ICU and their related confidence, https://gerrit.libreoffice.org/c/core/+/208899/1/tools/source/stream/stream.cxx#824) -- You are receiving this mail because: You are the assignee for the bug.
