On 26 June 2017 at 19:03, Eric Grange <egra...@glscene.org> wrote:

> No BOM = you have to fire a whole suite of heuristics or present the user
> with choices he/she will not understand.
>

Requiring heuristics to determine text encoding/codepage exists regardless
of whether BOM is used since the problem predates unicode altogether. Lets
consider the scenarios - as Simon enumerated we're roughly interested in
cases where the data stream begins with byte sequences:

(1) 0x00 0x00 0xFE 0xFF
(2) 0xFF 0xFE 0x00 0x00
(3) 0xFE 0xFF
(4) 0xFF 0xFE
(5) 0xEF 0xBB 0xBF
(6) anything else (and datastream is ASCII or UTF-8)
(7) anything else (and datastream is some random codepage)

In case 7 we have little choice but to invoke heuristics or defer to the
user, yes? For the first 5 cases we can immediately deduce some facts:

(1) -> almost certainly UTF-32BE, athough if NUL characters may be present
8-bit codepages are still a candidate
(2) -> almost certainly UTF-32LE, although if NUL characters may be present
UTF-16LE and 8-bit codepages are still a candidate
(3) -> likely UTF16-BE, but could be some other 8-bit codepage
(4) -> likely UTF16-LE, but could be some other 8-bit codepage
(5) -> almost certainly UTF-8, but could be some other 8-bit codepage

I observe that BOM never provides perfect confidence regarding the
encoding, although in practice I expect it would only fail on data
specifically designed to fool it.

I also suggest that the checks ought to be performed in the order listed,
to avoid categorising UTF-32LE text as UTF-16LE, and because the first 4
cases rule out [valid] UTF-8 data.


Now lets say we make an assumption that all text is UTF-8 until proven
otherwise. In case 6 we get lucky and everything works, and in case 7 we
find invalid characters and fall back to heuristics or the user to identify
the encoding.

In fact using this assumption we could dispense with the BOM entirely for
UTF-8 and drop case 5 from the list. So my question is, what advantage does
a BOM offer for UTF-8? What other cases can we identify with the
information it provides?

If you were going to jump straight from case 5 to case 7 in the absence of
a BOM it seems like you might aswell give UTF-8 a try since it and ASCII
are far and away the common case.


I'm sure I've simplified things with this description - have I missed
something crucial? Is the BOM argument about future proofing? Are we
worried about EBCDIC? Is my perspective too anglo-centric?

After 20 years, the choice is between doing the best in an imperfect world,
> or perpetuating the issue and blaming others.
>

By being scalable and general enough to represent all desired characters,
as I see it UTF-8 is not perpetuating any issues but rather offering an out
from historic codepage woes (by adopting it as the go-to interchange
format).

As Peter da Silva said:
> It’s not the UTF-8 storage that’s the mess, it’s the non-UTF-8 storage.

-Rowan
_______________________________________________
sqlite-users mailing list
sqlite-users@mailinglists.sqlite.org
http://mailinglists.sqlite.org/cgi-bin/mailman/listinfo/sqlite-users

Reply via email to