[EMAIL PROTECTED] skribis 2008-01-10 16:14 (-0800):
> +(Note that C<.bytes> is not guaranteed to be well-defined when the encoding
> +is unknown.

(This message is a mess; in my defense, it's 5:30 AM here. I just had to
respond, because I have the feeling Perl 6's unicode model is going
exactly the wrong way if I interpret this diff correctly.)

What if the encoding is known, but by accessing the .bytes level one
breaks the consistency?

Rather than a scheme where unicode text strings have an encoding
property, I think a scheme where unicode text strings are just unicode
text strings is better: the *binary* strings can have an encoding


* A Str is a sequence of codepoints, and provides grapheme/glyphs if
  requested. It does not have bytes, and the internal encoding does not
  show except through introspection. The internal encoding can
  theoretically change at Perl's will.
* A Buf is a sequence of bytes, not codepoints or characters of any
* A Buf with a defined .encoding:
  - does Str, with transparent decoding (with validity checking)
  - also, transparent encoding

my Str $foo = "H€łłø wöŕłđ";
my Buf $bar;
$bar.encoding = "utf-8";  # or however a decoding is declared
$bar = $foo;  # gets utf-8 encoded
$bar.bytes;   # [ "H", "\xE2", "\x82", "\xAC", ... ]
$bar.codes;   # [ "H", "€", "ł", ... ]
$foo.codes eqv $bar.codes  # true
$foo.bytes;   # Huh? What? Makes no sense -> fail

All byte-oriented mechanisms can have an encoding defined somehow:
filehandles, environment variables, Bufs, system call wrappers.

I think that would work much easier than giving Strs encoding
properties. When writing to a file, or a Buf, you're probably using a
lot of Strs, and it would make no sense to have them all encode
differently. Likewise, when reading from IO, a Buf, or anything
byte-oriented, the encoding will have to be known to decode it.

I fail to see how giving a Str any .bytes or .encoding might make sense:
these belong to byte strings, not text strings.

Making it easy to work with the internal byte buffer will take away
means of optimization, ease of changing our mind about the best
implementation encoding, and either security or performance (Either you
check the consistency every time you do something and everything is
slow, or you don't, and everything is potentially insecure when passed
on to other code.) Of course, the current internal encoding and byte
buffer should be accessible somehow, and maybe even writable for the
brave of heart, but IMO certainly not with the way too encouraging
.bytes thing - I'm tempted to call for .HOW.internal.

I think that a Buf with a defined encoding should check whether the data
is valid when reading, but a Str can skip this: Perl itself put the data
there, and trusts its own routines for much better performance.

Please, don't give Strs any byte semantics, but do give Bufs support for
transparent en-/decoding, and perhaps even unicode semantics.
Met vriendelijke groet,  Kind regards,  Korajn salutojn,

  Juerd Waalboer:  Perl hacker  <[EMAIL PROTECTED]>  <http://juerd.nl/sig>
  Convolution:     ICT solutions and consultancy <[EMAIL PROTECTED]>

Reply via email to