OK, this has turned into a long essay, so unless questions are really
addressed to me explicitly, I will try to avoid writing anything else
on this subject.
Here's my Jeremiad on Unicode. Take it for what it's worth.
"Johny Mattsson (EPA)" wrote:
> Part 1.1Type: Plain Text (text/plain)
| If we settle on wchar_t being 16bits, then we will still be forced to do
| UTF-7/8/16 to properly handle a random Unicode (or ISO/IEC 10646) string,
| since we must deal with that charming thing known as "surrogate pairs" (see
| section 3.7 of the Unicode standard v3.0). This again breaks the "one
| wchar_t == on character". When being forced to deal with Unicode, I much
| prefer working with 32bits, since that guarantees that I get a fixed length
| for each character. Admittedly, it is space inefficient to the Nth degree,
| but speedwise it is better.
ISO/IEC 10646-1 doesn't have any code points allocated above the
low 16 bits. It's the same as the Unicode 1.1 standard.
Unicode 3.0 throws a whole lot of dead languages into the mix,
or it tries to allocate seperate code points for non-existant
character sets, whose glyphs should be, according the Unicode
Philosophy that resulted in the controversial CJK unification,
unified with existing glyphs within the character set. Unicode,
after all, is a character set standard, not a font encoding
Unicode 3.x has not been ratified as an ISO/IEC standard, and it
may not ever be. So Unicode 3.x incursions above 16 bits are not
really a valid argument until Unicode 3.x is standardized in some
way other than administrative fiat by the Unicode Consortium
having published a new version to sell more books and justify its
continued existance to the people funding it.
Historically, I've really had a love/hate relationship with
When Unicode was originally designed, it was intentionally
designed to exclude fixed-cell rendering technologies: if
the font was pre-rendeered, you could not render characters
with ligatures intact.
Personally, I blame this in the fact that Taligent, the real
driving force behind the first Unicode standard, was an IBM
and Apple joint venture, and owed its pocket books to rendering
technologies like Display PostScript, which were direct
competitors with X Windows... and X Windows uses fixed cell
rendering technology, even when it's using TrueType fonts.
So when Unicode first came out, the "private use" areas were
not lare enough, nor sufficinetly near or interleaved with,
that of ligatured languages, like Tamil and Devengari, or
even Arabic and Hebrew.
There was a fundamental assumption that the rendering technology
would be disjoint from the encoding technology, and that the
cost, due to the arrangement of the "private use" areas, was to
be bourne in the rendering engine. And rendering engines where
that was not possible (e.g. X Windows) would just have to paint
pixels and eat the overhead in the applications (and they did;
you can install "xtamil" from ports and see how it works).
The Japanese *hate* Unicode. The primary reason for this hate
is, to be blunt, that Unicode is not a superset of JIS-208 or
JIS-208 + JIS-212; the secondary reason is that Japanese is as
nearly protectionist as French, and the CJK unification used
the Chinese dictionary order. There is a good reason for this,
however: Chinese dictionary order is capable of classifying
Japanese Ideograms. A simplification of this is that Chinese
dictionary classification is in "stroke, radical" order; thus
it is capable of classifying ideograms that "look like" they
are Chinese ideograms. The Japanese classification system is
not capable of doing this, and the Japanese have two widely
recognized classification systems for lexical ordering internal
to Japan, so it's not even possible to pick a "right order" if
you were to say "all the Japanese characters, *then* all the
In practice, this is a subject for academics who care about the
number of angels which can dance on the head of a pin. But it
has a slightly deeper protectionist agenda, as well. The
Japanese computer market, for a very long time, was not a
commoditized market. Perhaps the largest market share went to
the NEC-PC98 (indeed, there's explicit support in FreeBSD for
this machine). In such a market, it's possible to create
products which are non-commodity, and end up "owning" customers.
In addition, things like EUC encoding and XPG/4 are rarely
supported by non-Japanese software titles, which protects the
local software production market. MITI, in fact, has as one of
its mandates, the protection of a market for locally produced
Microsoft's introduction of Unicode, and the subsequent ability
of third party software written solely to support Microsoft
interfaces that used "oleString" and other wchar_t types
natively, meant that there was immediate support for Japanese
in these products. Microsoft broke down the wall that had
been built in order to protect local markets.
So, getting back to the main line of dicussion, with this
backround in hand:
| As for interoperability with Windows, it is clearly stated that the wchar_t
| is intended for internal usage only, and the various encoding schemes should
| be used when storing strings outside of a process. In reality this means
| that just about every Unicode capable application reads and writes in UTF-8
| or 7. This means that interoperability should not become an issue. If it
| really was expected to have been an issue, I'm sure the C++ standard would
| have mandated a specific width for wchar_t, which as far as I am aware they
Microsoft's OLESS (OLE Structured Storage), which is the storage
type that it uses for most Microsoft applications, these days,
has the capability of natively storing and retreiving OLE types,
Basically, this means that there is no conversion of the textual
data on its way in or out.
Your proposal, to take the phrase "internal use only" literally,
What it basically comes down to is the requirement for explicit
extra work to be done in order to support both l10n (localization)
and i18n (internationalization) in applications, rather than the
appilcations implicitly supporting them... as Microsoft applications
explicitly support them.
The net effect of doing this is that we will end up with a lot of
code, which, even if our hopes are realized, and it is 8-bit clean,
is missing a significant amount of engineering work, which would
be necessary for it to support languages which are not supported
by the 8-bit ISO-8859-1 (Latin-1) character set -- or whichever
8-bit character set has been selected as the local primary default
attribute assumed on otherwise unattributed text files.
Therefore, however the problem is handled, it is a good idea to
make sure that default applications, written without the ability
to explicitaly convert between internal (processing) and external
(storage) formats, still work for languages other than English.
| So, in the light of this, what would be the most appropriate choice? I
| haven't yet had a chance to explore what locales we support, but I would
| lean toward saying wchar_t == 32 bits, since this is future proof. If we
| later down the track are forced to go from 16 -> 32 due us supporting more
| of the asian locales, I foresee this causing _major_ breakage.
This ignores the interoperability issue which I originally
raised. I realize that "Windows" is a "dirty word" (though we
can all reread the above, and see how clever they are, even if
we want to pretend that they are technically inept), but... I
would like to see it possible to interoperate with third party
ELF libraries initially intended for use with Windows.
What this means is adoption of decisions similar to those made
by Windows in regards to minimum size assumptions with regard
to intrinsic types. Or even *exact* correspondance on size,
for values which may be externalized via librayr, COM, DCOM, or
some other marshalling facility that's going to assume that the
sizes of things marshalled in are going to be the same on the
You *could* "embrace and extend" what Microsoft has put out
there; however, lacking the ability to wield monopolistic power
in the marketplace, it's highly unlikely that your screams of
"My way is the right way!" will be heard over the steamroller
I guess a good point to make here is the size of a single
element in a "String" type in Java?
In any case, I rather expect that most of .NET is going to be
assuming 16 bit wchar's. Someone who knows, rather than just
"expecting" needs to speak up here.
In any case: expect to multiply your real storage requirements
by a factor of 2 for 16 bit Unicode, and a factor of 4, for 32
Unless you happen to be an English speaker who never bothers
setting the 8th bit on any of your text, UTF encoding is a raw
deal. Further, it will have a tendency to reduce the market
portability of software you write, no matter what, and without
a lot of extraordinary effort, expect that your code will only
be locally salable.
For Europeans, 8-bit clean won't save you any more. You will
end up having to take 2 characters to store any character in
the range 0x80-0xff. So no matter what, you European programmers
will be screwed by having storage encoding different from process
Look: I know that "Microsoft Invented it" is the kiss of death,
but isn't it possible to admit, just this once, that they maybe
had a good idea, and copy them?
To Unsubscribe: send mail to [EMAIL PROTECTED]
with "unsubscribe freebsd-current" in the body of the message