On 11/27/2013 08:53 AM, Jakob Ovrum wrote:
On Wednesday, 27 November 2013 at 16:18:34 UTC, Wyatt wrote:
I agree with the assertion that people SHOULD know how unicode works
if they want to work with it, but the way our docs are now is
off-putting enough that most probably won't learn anything. If they
know, they know; if they don't, the wall of jargon is intimidating
and hard to grasp (more examples up front of more things that you'd
actually use std.uni for). Even though I'm decently familiar with
Unicode, I was having trouble following all that (e.g. Isn't
"noe\u0308l" a grapheme cluster according to std.uni?). On the flip
side, std.utf has a serious dearth of examples and the relationship
between the two isn't clear.
I thought it was nice that std.uni had a proper terminology section,
complete with links to Unicode documents to kick-start beginners to
Unicode. It mentions its relationship with std.utf right at the top.
Maybe the first paragraph is just too thin, and it's hard to see the
big picture. Maybe it should include a small leading paragraph
detailing the three levels of Unicode granularity that D/Phobos
chooses; arrays of code units -> ranges of code points -> std.uni for
graphemes and algorithms.
Yes, please. While operations on single codepoints and characters
seem pretty robust (i.e. you can do lots of things with and to them),
it feels like it just falls apart when you try to work with strings.
It honestly surprised me how many things in std.uni don't seem to
work on ranges.
-Wyatt
Most string code is Unicode-correct as long as it works on code points
and all inputs are of the same normalization format; explicit
grapheme-awareness is rarely a necessity. By that I mean the most
common string operations, such as searching, getting a substring etc.
will work without any special grapheme decoding (beyond normalization).
The hiccups appear when code points are shuffled around, or the order
is changed. Apart from these rare string manipulation cases, grapheme
awareness is necessary for rendering code.
I would put things a bit more emphatically. The codepoint is analogous
to assembler, where the character is analogous to a high level language
(and the binary representation is analogous to a binary
representation). The desire is to make the characters easy to use in a
way that is cheap to do. To me this means that the highlevel language
(i.e., D) should make it easy to deal with characters, possible to deal
with codepoints, and you can deal with binary representations if you
really want to. (Also note the isomorphism between assembler code and
binary is matched by an isomorphism between codepoints and binary
representation.) To do this cheaply, D needs to know what kind of
normalization each string is in. This is likely to cost one byte per
string, unless there's some slack in the current representation.
But is this worth while? This is the direction that things will
eventually go, but that doesn't really mean that we need to push them in
that direction today. But if D had a default normalization that
occurred during i/o operations, to cost of the normalization would
probably be lost during the impedance matching between RAM and storage.
(Again, however, any default requires the ability to be overridden.)
Also, of course, none of this will be of any significance to ASCII.
--
Charles Hixson
