On 2011-01-14 09:34:55 -0500, "Steven Schveighoffer"
<[email protected]> said:
On Fri, 14 Jan 2011 08:59:35 -0500, spir <[email protected]> wrote:
The point is not playing like that with Unicode flexibility. Rather
that composite characters are just normal thingies in most languages
of the world. Actually, on this point, english is a rare exception
(discarding letters imported from foreign languages like french 'à');
to the point of beeing, I guess, the only western language without any
diacritic.
Is it common to have multiple modifiers on a single character?
Not in my knowledge. But I rarely deal with non-latin texts, there's
probably some scripts out there that takes advantage of this.
The problem I see with using decomposed canonical form for strings is
that we would have to return a dchar[] for each 'element', which
severely complicates code that, for instance, only expects to handle
English.
Actually, returning a sliced char[] or wchar[] could also be valid.
User-perceived characters are basically a substring of one or more code
points. I'm not sure it complicates that much the semantics of the
language -- what's complicated about writing str.front == "a" instead
of str.front == 'a'? -- although it probably would complicate the
generated code and make it a little slower.
In the case of NSString in Cocoa, you can only access the 'characters'
in their UTF-16 form. But everything from comparison to search for
substring is done using graphemes. It's like they implemented
specialized Unicode-aware algorithms for these functions. There's no
genericness about how it handles graphemes.
I'm not sure yet about what would be the right approach for D.
I was hoping to lazily transform a string into its composed canonical
form, allowing the (hopefully rare) exception when a composed character
does not exist. My thinking was that this at least gives a useful
string representation for 90% of usages, leaving the remaining 10% of
usages to find a more complex representation (like your Text type).
If we only get like 20% or 30% there by making dchar the element type,
then we haven't made it useful enough.
Either way, we need a string type that can be compared canonically for
things like searches or opEquals.
I wonder if normalized string comparison shouldn't be built directly in
the char[] wchar[] and dchar[] types instead. Also bring the idea above
that iterating on a string would yield graphemes as char[] and this
code would work perfectly irrespective of whether you used combining
characters:
foreach (grapheme; "exposé") {
if (grapheme == "é")
break;
}
I think a good standard to evaluate our handling of Unicode is to see
how easy it is to do things the right way. In the above, foreach would
slice the string grapheme by grapheme, and the == operator would
perform a normalized comparison. While it works correctly, it's
probably not the most efficient way to do thing however.
--
Michel Fortin
[email protected]
http://michelf.com/
