On Sunday, 13 October 2013 at 13:14:59 UTC, nickles wrote:
This is simply wrong. All strings return number of codeunits.
And it's only UTF-32 where codepoint (~ character) happens to
fit into one codeunit.
I do not agree:
writeln("säд".length); => 5 chars: 5 (1 + 2 [C3A4] +
2 [D094], UTF-8)
writeln(std.utf.count("säд")) => 3 chars: 5 (ibidem)
writeln("säд"w.length); => 3 chars: 6 (2 x 3, UTF-16)
writeln("säд"d.length); => 3 chars: 12 (4 x 3, UTF-32)
This is not consistent - from my point of view.
This is not a single inconsistency here.
First of all, typeof("säд") yileds string type (immutable char)
while typeof(['s', 'ä', 'д']) yileds neither char[], nor wchar[],
nor even dchar[] but int[]. In this case D is close to C which
also treats character literals as integer type. Secondly,
character arrays are only one who have two kinds of array
literals - usual [item. item, item] and special "blah", as you
see there is no correspondence between them.
If you try char[] x = cast(char[])['s', 'ä', 'д'] then length
would be indeed 3 (but don't use it - it is broken).
In D dynamic array is at binary level represented as struct {
void *ptr; size_t length; }. When you perform some operations on
dynamic arrays they are implemented by compiler as calls to
runtime functions. However, during runtime it is impossible to do
something useful on arrays for which there is only information
about address of beginning and total elements (this is a source
of other problems in D). To handle this, compiler generates and
passes as separate argument "TypeInfo" to runtime functions.
TypeInfo contains some data, most relevant here is size of the
element.
What happens is follows. Compiler recognizes that "säд" should be
string literal and encoded as UTF-8
(http://dlang.org/lex.html#DoubleQuotedString), so element type
should be char, which requires to have 5 elements in array. So,
at runtime an object "säд" is treated as array of 5 elements each
having 1 byte per element.
Basically string (and char[]) plays dual role in the language -
on the one hand, it is array of elements having strictly 1 byte
size by definition, on the other hand D tries to use it as
'generic' UTF type for which size is not fixed. So, there is
contradiction - in source code such strings are viewed by
programmer as some abstract UTF string, but druntime views it as
5 byte array. In my view, trouble begins when "säд" is internally
casted to char (which is no better than int[] x = [3.14, 5.6]).
And indeed, char[] x = ['s', 'ä', 'д'] is refused by language, so
there is great inconsistency here.
By the way, UTF definition is irrelevant here, this is pure
implementation issue (I think it is design fault).
