Marcin 'Qrczak' Kowalczyk wrote:
> Lars Kristan <[EMAIL PROTECTED]> writes:
>
> > Quite close. Except for the fact that:
> > * U+EE93 is represented in UTF-32 as 0x0000EE93
> > * U+EE93 is represented in UTF-16 as 0xEE93
> > * U+EE93 is represented in UTF-8 as 0x93 (_NOT_ 0xEE 0xBA 0x93)
>
> Then it would be impossible to represent sequences like
> U+EEEE U+EEBA U+EE93 in UTF-8, and conversion UTF-32 -> UTF-8
> -> UTF-32
> would not round-trip.
>
> Concatenation of UTF-8-encoded strings would not be equivalent to
> UTF-8-encoding of the concatenation of code points.
>
I am well aware of that fact. And have also stated so in my original mail:
--- quote ---
The other name for this is roundtripping. Currently, Unicode allows a roundtrip UTF-16=>UTF-8=>UTF-16. For any data. But there are several reasons why a UTF-8=>UTF-16(32)=>UTF-8 roundtrip is more valuable, even if it means that the other roundtrip is no longer guaranteed:
--- end quote ---
Let me go a bit further. A UTF-16=>UTF-8=>UTF-16 roundtrip is only required for valid codepoints other than the surrogates. But it also works for surrogates unless you explicitly and intentionally break it. One can choose to do so, or one can choose not to do so. I would opt for the latter (i.e. keep it working), according to my statement (in the thread "When to validate") that validation should be separated from other processing, where possible. Due to performance and practicality reasons a conversion function can contain validation, but if it does, this behavior should be switchable.
A UTF-32=>UTF-8=>UTF-32 roundtrip is similar, except that 16-8-16 works even with concatenation, while 32-8-32 can be broken with concatenation.
A UTF-32=>UTF-16=>UTF-32 roundtrip is not guaranteed (still talking about surrogates), even without concatenation.
All this is known and presents no problems, or - only problems that can be kept under control. So, by introducing another set of 128 'surrogates', we don't get a new type of a problem, just another instance of a well known one.
On the other hand, UTF-8=>UTF-16=>UTF-8 as well as UTF-8=>UTF-32=>UTF-8 can be both achieved, with no exceptions. This is something no other roundtrip can offer at the moment. This fact alone should be enough to make us want to have it.
On top of it, I repeatedly stressed that it is UTF-8 data that has the highest probablility of any of the following:
* contains portions that are not UTF-8
* is not really UTF-8, but user has UTF-8 set as default encoding
* is not really UTF-8, but was marked as such
* a transmission error not only changes data but also creates invalid sequences
UTF-16 has a much smaller possibility of being 'corrupted'. There are no legacy encodings that could creep it. The only thing that could creep in would be LE/BE mixup, any 8-bit data or 32-bit data. But that doesn't happen, because those mixups don't work even with ASCII text and are quickly detected and prevented.
So, not only that UTF-32=>UTF-8=>UTF-32 (or UTF-16=>UTF-8=>UTF-16) don't roundtrip as-is, it doesn't really matter whether they do or don't. As long as they do for valid Unicode strings, containing no surrogates.
Which does not mean we do not need to define how the conversions behave for the surrogates (and I don't mean the values of surrogates in a UTF-16). But that is another issue.
> This is broken.
So, just as much as UTF-16 is broken.
