OK, I think I get 'use byte' vs. 'use utf8' vs. neither.  But, there
are still locales.  To illustrate the issues, let me get concrete here
for a minute.  (Please, no construction jokes.)

Consider the apparently straightforward:
    @a = sort @b;

AFAIK, in the absence of pragmas, Perl 5.6 will compare the elements
of @b character-by-character, regardless of whether the characters in
question have one- or multi-byte representations.  This behavior
should be well-defined even if some elements of @b are Unicode and
some aren't.  (I'm glad that this is the default.)

Now, from the sublime to the ridiculous:

C locales define various characters characteristics.  But they are so
limited as to be very difficult to use.  More to the point, they are
defined in terms of the C type 'char', so there can be no support for
multi-byte character encodings unless your 'char' type is larger than
one byte.  Furthermore, C makes no provision for cross-locale
processing -- you can't meaningfully collate e.g. English and Hebrew.

Therefore, if C<sort> is told to use locales, must consider all
strings to contain characters from the current (run-time) locale.

Now, comes the key bit: It's entirely possible for characters, even
characters considered to be from the current locale, to be encoded in
a number of ways.  In other words: STRING ENCODING AND CHARACTER SET
ARE ORTHOGONAL.  So I propose that OPs' string encoding states and
character set states be orthogonal in a user-visible way.

(Granted, C locales are limited to one-byte characters, so anything
with ord() > 255 has no place in a locale-charset string.  But Topaz
uses C++, and C++ locales apply not only to 'char's, but also to
'wchar_t's -- wide characters.  AFAIK, there is no technical obstacle
to a C++ implementation's providing Unicode-compatible locales.
Besides, some intrepid user may want to use a non-Unicode character
set while obeying each string's current encoding.)

So: The _string_encoding_ state of each OP must be one of these:

  0. the default -- follow each string's current encoding
  1. "use byte"  -- all strings are one-byte
  2. "use utf8"  -- all strings are UTF-8 (*not* necessarily Unicode!)

And the _character_set_ state of each OP must one of these:

  0. the default  -- characters are Latin-1, UTF-8 is Unicode
  1. "use locale" -- characters are $ENV{LANG} (set at runtime)

If you just want to stop here, then please consider the above as a
proposed specification for the interaction of UTF-8 and locales.


Seeing the above list of pragmas triggers my generalization reflex.
So, how about this:

  0. C<no encoding>         == the default
  1. C<use encoding 'utf8'> == C<use utf8>
  2. C<use encoding 'byte'> == C<use byte>

Combined with this:

  0. C<no charset>           == the default
  1. C<use charset 'locale'> == C<use locale>

This interface would also provide a hook for any encodings we might
support in future:
  use encoding 'byte2big'; == force two-byte big-endian characters,
                                without forcing their charset
  use encoding 'byte4big'; == force four-byte big-endian characters
  use charset 'iso10646';  == force ISO 10646 (Unicode superset)

So, what do you think?
Chip Salzenberg          - a.k.a. -           <[EMAIL PROTECTED]>
        "He's Mr. Big of 'Big And Tall' fame."  // MST3K

Reply via email to