Re: unicode confusion--An answer

[Charles Hixson via Digitalmars-d-learn]

I was writing my output to two different files.  Only one of them was 
set to utf-8, the other must have been some other encoding, because when 
I set the encoding to utf-8 everything cleared up.

On 04/04/2016 04:04 PM, Charles Hixson via Digitalmars-d-learn wrote:
Well, at least I think that it's unicode confusion.  When a store 
values into a string (in an array of structs) and then compare it 
against itself, it compares fine, and if I write it out at that point 
it writes out fine.  And validate says it's good unicode.

But later...
valid = true, len = 17, wrd = true , cnt =     2, txt = Gesammtabentheuer
valid = true, len = 27, wrd = true , cnt =     1, txt = 
νεφεληγερέτης
valid = true, len = 17, wrd = true , cnt =     1, txt = ζητοῦσιν
valid = true, len = 36, wrd = true , cnt =     1, txt = 
αἱμορροϊδοκαύστης
valid = true, len = 18, wrd = true , cnt =     2, txt = 
δυνηθώμεν
valid = true, len = 20, wrd = true , cnt =     1, txt = 
προσκρούσω
valid = true, len = 20, wrd = true , cnt =     1, txt = 
σκοτωμένην
valid = true, len = 18, wrd = true , cnt =     1, txt = 
ἀγαπητοί
valid = true, len = 28, wrd = true , cnt =     1, txt = 
הַֽמְזִמָּתָהּ
valid = true, len = 19, wrd = true , cnt =     1, txt = 
Τυρρηνικά
valid = true, len = 17, wrd = true , cnt =     2, txt = IODOHYDRARGYRATIS
valid = true, len = 21, wrd = true , cnt =     1, txt = 
χοινικίσιν
valid = true, len = 17, wrd = true , cnt =     1, txt = Spectrophotometer
valid = true, len = 26, wrd = true , cnt =     1, txt = 
αἰνιττόμενοι
valid = true, len = 70, wrd = 
true , cnt =     1, txt = ΓΗΣΠ
ΛΕΘΡΑΔΙΣΧΙΛΙΑΕΡΓΑΣΙΜΟΥΑΠΟΤΗΣΟΜΟ
valid = true, len = 18, wrd = true , cnt =     1, txt = 
μικρότατα
valid = true, len = 23, wrd = true , cnt =     1, txt = 
ἀποπάτησίν
valid = true, len = 18, wrd = true , cnt =     1, txt = 
מוֹקְשֵׁי
valid = true, len = 17, wrd = true , cnt =     1, txt = διαμένων
     . . . (etc. for 39599 lines)
(And it looks worse than that, actually, because control characters 
aren't coming through).
I think the originals were usually greek letters due to an earlier 
test (why there should be so many greek words I don't know...but if 
they're there I want them to be handled properly), but the corrupted 
text is such a small part of the original file that I can't be 
certain.  Valid = true means that it passed string validates right 
before being printed.  wrd = true means that the only characters in it 
should be isAlpha, hyphen, apostrophe, or underscore.  cnt = n means 
that it was detected n times in the dataset (of 8013 text files). And 
the string in each struct is only written once in the execution of the 
program.

I was scanning the dataset looking to see what long words were 
valid...I didn't expect THIS at all.  And as you can see from, e.g., 
"Spectrophotometer", ASCII values don't seem to be damaged at all.

FWIW, I was expecting to encounter an occasional Greek, French, or 
Chinese word...but nothing like this.  I'd think it was the conversion 
from string to dchar[] and back that was the problem, but when I test 
immediately after I know I've written to the string everything looks 
right.  So I'm guessing it's something about how unicode is handled.