On Fri, Apr 30, 2004 at 10:58:19PM +0700, Martin Hosken wrote:
IIRC AL32UTF8 was introduced at the behest of Oracle (a voting member of Unicode) because they were storing higher plane codes using the surrogate pair technique of UTF-16 mapped into UTF-8 (i.e. resulting in 2 UTF-8 chars or 6 bytes) rather than the correct UTF-8 way of a single char of 4+ bytes. There is no real trouble doing it that way since anyone can convert between the 'wrong' UTF-8 and the correct form. But they found that if you do a simple binary based sort of a string in AL32UTF8 and compare it to a sort in true UTF-8 you end up with a subtly different order. On this basis they made request to the UTC to have AL32UTF8 added as a kludge and out of the kindness of their hearts the UTC agreed thus saving Oracle from a whole heap of work. But all are agreed that UTF-8 and not AL32UTF8 is the way forward.
Um, now you've confused me.
The Oracle docs say "In AL32UTF8, one supplementary character is represented in one code point, totalling four bytes." which you say is "correct UTF-8 way". So the old Oracle ``UTF8'' charset is what's now called "CESU-8" and what Oracle call ``AL32UTF8'' is the "correct UTF-8 way".
> So did you mean CESU-8 when you said AL32UTF8?
I guess so.
Thank you for reminding me of this. I used to know that, but forgot it and was about to write my colleague to use 'UTF8' (instead of 'AL32UTF8') when she creates a database with Oracle for our project.
Oracle is notorious for using 'incorrect' and confusing character encoding names. Their 'AL32UTF8' is the true and only UTF-8 while __their__ 'UTF8' is CESU-8 (a beast that MUST be confined within Oracle and MUST NOT be leaked out to the world at large. Needless to say, it'd be even better had it not been born.)
Oracle has no execuse whatsoever for failing to get their 'UTF8' right in the first place because Unicode had been extended beyond BMP a long time before they introduced UTF8 into their product(s) (let alone the fact that ISO 10646 had non-BMP planes from the very beginning in 1980's and that UTF-8 was devised to cover the full set of ISO 10646) However, they failed and in their 'UTF8', a single character beyond BMP was (and still is) encoded as a pair of 3-byte representations of surrogate code points. Apparently for the sake of backward compatibility (I wonder how many instances of Oracle databases existed with non-BMP characters stored in their 'UTF8' when they decided to follow this route), they decided to keep the designation 'UTF8' for CESU-8 and came up with a new designation 'AL32UTF8' for the true and only UTF-8.
Jungshik
